Nov 03 14:10:49.840147 ip-10-0-130-191 systemd[1]: Starting Kubernetes Kubelet... Nov 03 14:10:50.184145 ip-10-0-130-191 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 14:10:50.184145 ip-10-0-130-191 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 14:10:50.184145 ip-10-0-130-191 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 14:10:50.184145 ip-10-0-130-191 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 14:10:50.187898 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187800 2566 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:50.187898 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187897 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:50.187898 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187901 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187905 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187908 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187910 2566 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187913 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187915 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187918 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187921 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187924 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187926 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187930 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187933 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187935 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187938 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187941 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187943 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187946 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187948 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187951 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187954 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187956 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:50.187991 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187958 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187961 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187964 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187966 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187969 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187976 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187979 2566 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187981 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187985 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187989 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187992 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187995 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.187998 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188001 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188003 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188006 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188009 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188011 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188014 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:50.188474 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188017 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188019 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188022 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188025 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188028 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188031 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188033 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188036 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188038 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188041 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188044 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188047 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188050 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188053 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188055 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188058 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188060 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188063 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188066 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188068 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:50.188945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188071 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188074 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188077 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188079 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188082 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188084 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188087 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188089 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188092 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188094 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188097 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188099 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188102 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188104 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188107 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188109 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188112 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188115 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188118 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:50.189426 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188120 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188123 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188125 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188128 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188130 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188133 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188136 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188138 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188141 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188144 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188146 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188149 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188154 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188158 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188161 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188164 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188166 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188169 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188171 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:50.189895 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188173 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188176 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188178 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188181 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188183 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.188186 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189066 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189074 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189077 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189079 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189082 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189086 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189088 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189091 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189094 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189096 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189099 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189102 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189104 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189107 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:50.190346 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189110 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189113 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189116 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189118 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189121 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189123 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189126 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189129 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189131 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189134 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189136 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189138 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189141 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189144 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189146 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189149 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189151 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189154 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189156 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:50.190829 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189158 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189161 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189164 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189166 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189169 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189172 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189174 2566 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189177 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189180 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189182 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189185 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189188 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189190 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189193 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189196 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189199 2566 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189201 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189204 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189206 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189209 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:50.191290 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189211 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189214 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189216 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189218 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189221 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189223 2566 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189226 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189229 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189231 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189234 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189236 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189239 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189241 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189244 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189247 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189249 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189252 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189254 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189257 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189260 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:50.191769 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189263 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189265 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189268 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189270 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189272 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189275 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189278 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189280 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189283 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189285 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189288 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189290 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189292 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189295 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189297 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189300 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189302 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189305 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189307 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:50.192266 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189310 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189323 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189326 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189329 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189332 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189334 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189337 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189340 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189342 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189345 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189347 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189350 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189353 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189358 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189881 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189887 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189890 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189894 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189897 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:50.192724 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189900 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189902 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189905 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189907 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189910 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189913 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189917 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189921 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189925 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189929 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189933 2566 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189937 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189941 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189945 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189949 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189952 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189954 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189957 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189959 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189965 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:50.193180 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189968 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189971 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189974 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189990 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.189994 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190004 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190007 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190010 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190013 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190015 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190018 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190023 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190025 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190028 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190030 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190033 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190035 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190040 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190044 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:50.193701 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190047 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190049 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190052 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190054 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190057 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190059 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190062 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190064 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190067 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190069 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190072 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190074 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190078 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190080 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190083 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190085 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190089 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190093 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:50.194176 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190095 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190104 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190107 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190109 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190112 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190114 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190118 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190121 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190123 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190126 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190128 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190131 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190133 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190136 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190138 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190140 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190143 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190145 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190148 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190151 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:50.194643 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190153 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190156 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190158 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190160 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190163 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190165 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190169 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190171 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190174 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190176 2566 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190178 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190181 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190183 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190186 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190188 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190192 2566 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190195 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190197 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190200 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190203 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:50.195133 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190205 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190208 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190210 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.190213 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191099 2566 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191112 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191120 2566 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191125 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191130 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191133 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191138 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191142 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191145 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191148 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191151 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191155 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191158 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191161 2566 flags.go:64] FLAG: --cgroup-root="" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191165 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191168 2566 flags.go:64] FLAG: --client-ca-file="" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191171 2566 flags.go:64] FLAG: --cloud-provider="external" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191174 2566 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191178 2566 flags.go:64] FLAG: --cluster-domain="" Nov 03 14:10:50.195704 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191181 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191184 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191187 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191190 2566 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191195 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191198 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191201 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191205 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191208 2566 flags.go:64] FLAG: --contention-profiling="false" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191211 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191214 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191217 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191220 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191224 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191227 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191230 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191233 2566 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191236 2566 flags.go:64] FLAG: --enable-server="true" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191239 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191251 2566 flags.go:64] FLAG: --event-burst="100" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191255 2566 flags.go:64] FLAG: --event-qps="50" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191258 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191261 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191264 2566 flags.go:64] FLAG: --eviction-hard="" Nov 03 14:10:50.196290 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191272 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191275 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191278 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191281 2566 flags.go:64] FLAG: --eviction-soft="" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191284 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191287 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191290 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191293 2566 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191296 2566 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191299 2566 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191302 2566 flags.go:64] FLAG: --feature-gates="" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191306 2566 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191309 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191313 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191316 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191320 2566 flags.go:64] FLAG: --healthz-port="10248" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191323 2566 flags.go:64] FLAG: --help="false" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191326 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191329 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191332 2566 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191335 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191338 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191342 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191344 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 14:10:50.196899 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191347 2566 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191350 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191353 2566 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191356 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191360 2566 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191363 2566 flags.go:64] FLAG: --kube-reserved="" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191365 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191368 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191371 2566 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191374 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191377 2566 flags.go:64] FLAG: --lock-file="" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191379 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191382 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191385 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191391 2566 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191394 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191397 2566 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191399 2566 flags.go:64] FLAG: --logging-format="text" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191402 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191406 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191409 2566 flags.go:64] FLAG: --manifest-url="" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191412 2566 flags.go:64] FLAG: --manifest-url-header="" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191417 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191420 2566 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191424 2566 flags.go:64] FLAG: --max-pods="110" Nov 03 14:10:50.197473 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191427 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191430 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191433 2566 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191436 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191439 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191442 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191444 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191452 2566 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191455 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191458 2566 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191461 2566 flags.go:64] FLAG: --pod-cidr="" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191463 2566 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191466 2566 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191469 2566 flags.go:64] FLAG: --pods-per-core="0" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191472 2566 flags.go:64] FLAG: --port="10250" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191475 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191478 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bcaa5720b29ec280" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191480 2566 flags.go:64] FLAG: --qos-reserved="" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191483 2566 flags.go:64] FLAG: --read-only-port="10255" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191486 2566 flags.go:64] FLAG: --register-node="true" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191489 2566 flags.go:64] FLAG: --register-with-taints="" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191493 2566 flags.go:64] FLAG: --registry-burst="10" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191495 2566 flags.go:64] FLAG: --registry-qps="5" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191498 2566 flags.go:64] FLAG: --reserved-cpus="" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191501 2566 flags.go:64] FLAG: --reserved-memory="" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191505 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 14:10:50.198341 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191508 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191511 2566 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191514 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191517 2566 flags.go:64] FLAG: --runonce="false" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191520 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191523 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191526 2566 flags.go:64] FLAG: --seccomp-default="false" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191529 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191533 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191537 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191540 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191543 2566 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191546 2566 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191549 2566 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191552 2566 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191555 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191558 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191561 2566 flags.go:64] FLAG: --system-cgroups="" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191563 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191568 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191571 2566 flags.go:64] FLAG: --tls-cert-file="" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191574 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191578 2566 flags.go:64] FLAG: --tls-min-version="" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191581 2566 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191584 2566 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 14:10:50.199440 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191587 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191590 2566 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191593 2566 flags.go:64] FLAG: --v="2" Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191597 2566 flags.go:64] FLAG: --version="false" Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191601 2566 flags.go:64] FLAG: --vmodule="" Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191605 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.191608 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191751 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191757 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191760 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191763 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191766 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191769 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191771 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191774 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191778 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191781 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191783 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191787 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191791 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:50.200485 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191793 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191796 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191798 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191801 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191803 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191806 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191808 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191811 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191813 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191816 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191818 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191821 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191824 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191826 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191829 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191831 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191833 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191836 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191838 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191853 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:50.201440 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191858 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191860 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191863 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191865 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191868 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191871 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191874 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191878 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191882 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191885 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191888 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191890 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191893 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191896 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191898 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191901 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191903 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191906 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191908 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:50.202452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191911 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191913 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191916 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191918 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191921 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191923 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191926 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191928 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191931 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191933 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191936 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191938 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191941 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191945 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191947 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191950 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191952 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191955 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191957 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191960 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:50.202977 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191964 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191966 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191969 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191971 2566 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191974 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191976 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191979 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191982 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191984 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191987 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191989 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191992 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.191995 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192000 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192004 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192008 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192012 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192016 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192019 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:50.203669 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192023 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192026 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192029 2566 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192032 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192035 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192037 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192041 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192043 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192046 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192048 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192051 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192053 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192056 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192061 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192063 2566 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.192091 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 14:10:50.204465 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192136 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192140 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192143 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192146 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192149 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192151 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192154 2566 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192157 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192159 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192162 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192164 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192167 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192169 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192171 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192174 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192176 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192179 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192181 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192184 2566 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192187 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:50.205029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192189 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192191 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192194 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192199 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192201 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192204 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192206 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192209 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192211 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192214 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192219 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192223 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192226 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192229 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192232 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192234 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192237 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192239 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192241 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:50.205661 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192244 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192246 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192249 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192251 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192254 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192256 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192258 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192261 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192263 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192266 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192268 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192271 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192273 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192276 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192279 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192281 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192285 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192288 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192290 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192294 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:50.206137 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192296 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192301 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192306 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192310 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192313 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192315 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192318 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192320 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192323 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192325 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192328 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192330 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192333 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192336 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192338 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192341 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192343 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192346 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192348 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:50.206807 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192350 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192353 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192356 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192358 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192361 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192363 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192365 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192368 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192370 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192374 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192377 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192379 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192382 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192384 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192386 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192390 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192392 2566 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192395 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192398 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:50.207470 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192401 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192403 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192406 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192409 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192411 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192414 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192416 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192419 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.192421 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.200364 2566 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.200383 2566 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.200406 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.200414 2566 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200504 2566 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200513 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200518 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200524 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200529 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200533 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:50.207984 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200537 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200541 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200546 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200550 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200554 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200558 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200563 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200567 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200571 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200575 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200580 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200584 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200590 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200594 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200598 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200602 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200606 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200611 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200615 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:50.208488 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200620 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200624 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200628 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200632 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200637 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200641 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200645 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200650 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200654 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200658 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200662 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200667 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200671 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200675 2566 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200679 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200683 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200687 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200691 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200696 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200700 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:50.209014 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200704 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200708 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200712 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200716 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200720 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200724 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200729 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200733 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200738 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200742 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200747 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200751 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200755 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200759 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200764 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200768 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200771 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200775 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200779 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:50.209498 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200784 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200788 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200793 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200797 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200802 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200806 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200810 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200814 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200818 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200823 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200827 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200831 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200836 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200859 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200864 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200868 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200872 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200877 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200881 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200885 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:50.209964 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200890 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200895 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200898 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200903 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200908 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200912 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200918 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200924 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200931 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200937 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200942 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200948 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200953 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200957 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200962 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200966 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200971 2566 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200975 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200979 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:50.210435 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200983 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200987 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.200992 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.201036 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201269 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201278 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201284 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201288 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201292 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201297 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201301 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201305 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201309 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201321 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201327 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:50.210903 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201331 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201338 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201345 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201350 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201354 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201359 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201364 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201368 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201373 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201377 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201382 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201386 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201390 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201394 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201398 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201402 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201406 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201411 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201415 2566 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:50.211251 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201419 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201423 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201427 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201431 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201435 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201439 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201443 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201446 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201450 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201454 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201457 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201461 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201466 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201471 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201476 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201480 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201484 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201488 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201493 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201497 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:50.211692 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201501 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201505 2566 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201509 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201513 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201517 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201521 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201526 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201530 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201534 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201538 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201542 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201546 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201550 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201555 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201560 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201564 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201569 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201573 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201577 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:50.212263 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201581 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201586 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201590 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201594 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201598 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201602 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201606 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201611 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201616 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201620 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201624 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201628 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201632 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201637 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201641 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201644 2566 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201649 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201653 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201658 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:50.212721 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201662 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201666 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201670 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201674 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201678 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201682 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201686 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201693 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201699 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201703 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201708 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201712 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201716 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201720 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201724 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201728 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201732 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:50.213226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201736 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.201776 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201833 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201857 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201863 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201867 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201871 2566 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201875 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201879 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201883 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201887 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201890 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201894 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201898 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201902 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201905 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:50.213622 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201909 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201913 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201916 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201921 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201924 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201928 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201932 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201935 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201939 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201943 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201946 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201950 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201954 2566 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201958 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201961 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201966 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201969 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201973 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201977 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201981 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:50.214029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201984 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201990 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201994 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.201998 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202002 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202006 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202010 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202015 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202019 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202023 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202027 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202031 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202038 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202043 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202047 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202051 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202055 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202059 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202063 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:50.214503 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202068 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202072 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202076 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202080 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202085 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202089 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202093 2566 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202097 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202102 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202106 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202110 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202114 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202119 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202123 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202127 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202133 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202137 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202143 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202150 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:50.214972 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202155 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202161 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202165 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202169 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202174 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202178 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202183 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202188 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202192 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202196 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202200 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202205 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202209 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202213 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202217 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202222 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202225 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202229 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202234 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202238 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:50.215452 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202242 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202248 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202255 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202259 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202264 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202270 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202275 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202279 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202285 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202295 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202299 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202303 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202307 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:50.202312 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.202401 2566 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.205341 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.206949 2566 server.go:1014] "Starting client certificate rotation" Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.207390 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 14:10:50.215945 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.207435 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 14:10:50.216361 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.209486 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 14:10:50.216361 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.212074 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 14:10:50.223077 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.223060 2566 log.go:25] "Validated CRI v1 runtime API" Nov 03 14:10:50.226359 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.226344 2566 log.go:25] "Validated CRI v1 image API" Nov 03 14:10:50.227218 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.227191 2566 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 14:10:50.229993 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.229973 2566 fs.go:135] Filesystem UUIDs: map[564230ad-cbaa-4291-a44a-18780785af32:/dev/nvme0n1p3 6db23011-393a-4b9f-86b9-83631f4fa445:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 14:10:50.230056 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.229993 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 14:10:50.235084 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.234971 2566 manager.go:217] Machine: {Timestamp:2026-11-03 14:10:50.233671476 +0000 UTC m=+0.292164759 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099176 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21bbbd861ef3145e25fe3ecbb4855c SystemUUID:ec21bbbd-861e-f314-5e25-fe3ecbb4855c BootID:fb6faccc-3d96-4c98-a158-474bef4f7817 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f9:8c:12:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f9:8c:12:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:eb:b1:cd:d2:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 14:10:50.235084 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.235084 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 14:10:50.235210 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.235199 2566 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 14:10:50.237750 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.237725 2566 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 14:10:50.237914 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.237753 2566 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-191.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 14:10:50.238489 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.238479 2566 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 14:10:50.238534 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.238492 2566 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 14:10:50.238534 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.238510 2566 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 14:10:50.238534 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.238523 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 14:10:50.239315 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.239305 2566 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 14:10:50.242495 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.242482 2566 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 14:10:50.242615 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.242606 2566 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 14:10:50.243233 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.243224 2566 kubelet.go:493] "Attempting to sync node with API server" Nov 03 14:10:50.243268 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.243237 2566 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 14:10:50.243268 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.243247 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 14:10:50.243268 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.243256 2566 kubelet.go:405] "Adding apiserver pod source" Nov 03 14:10:50.243268 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.243264 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 14:10:50.243553 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.243541 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 14:10:50.245668 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.245646 2566 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 14:10:50.251229 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.251211 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 14:10:50.251918 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.251905 2566 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 14:10:50.251962 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.251936 2566 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 14:10:50.252085 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252075 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 14:10:50.252120 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252090 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 14:10:50.252120 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252096 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 14:10:50.252120 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252101 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 14:10:50.252120 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252107 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 14:10:50.252120 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252112 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 14:10:50.252767 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252757 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 14:10:50.252801 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252769 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 14:10:50.252801 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252777 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 14:10:50.252801 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252782 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 14:10:50.252801 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252794 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 14:10:50.252990 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252805 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 14:10:50.252990 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252827 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 14:10:50.252990 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.252833 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 14:10:50.256465 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.256452 2566 server.go:1263] "Started kubelet" Nov 03 14:10:50.256575 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.256531 2566 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 14:10:50.256686 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.256536 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 14:10:50.256801 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.256772 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 14:10:50.257257 ip-10-0-130-191 systemd[1]: Started Kubernetes Kubelet. Nov 03 14:10:50.257812 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.257797 2566 server.go:317] "Adding debug handlers to kubelet server" Nov 03 14:10:50.257918 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.257837 2566 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 14:10:50.260905 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.260885 2566 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 14:10:50.262835 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.262816 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 14:10:50.263773 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.263759 2566 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 14:10:50.263911 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.263894 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:50.264376 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.264359 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 14:10:50.264376 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.264359 2566 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 14:10:50.264505 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.264383 2566 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 14:10:50.264505 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.264464 2566 reconstruct.go:97] "Volume reconstruction finished" Nov 03 14:10:50.264505 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.264474 2566 reconciler.go:29] "Reconciler: start to sync state" Nov 03 14:10:50.264588 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.264570 2566 factory.go:55] Registering systemd factory Nov 03 14:10:50.264588 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.264581 2566 factory.go:223] Registration of the systemd container factory successfully Nov 03 14:10:50.264789 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.264778 2566 factory.go:153] Registering CRI-O factory Nov 03 14:10:50.264867 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.264791 2566 factory.go:223] Registration of the crio container factory successfully Nov 03 14:10:50.264867 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.264838 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 14:10:50.264949 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.264870 2566 factory.go:103] Registering Raw factory Nov 03 14:10:50.264949 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.264881 2566 manager.go:1199] Started watching for new ooms in manager Nov 03 14:10:50.265690 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.265678 2566 manager.go:319] Starting recovery of all containers Nov 03 14:10:50.272646 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.272620 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 14:10:50.275167 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.275151 2566 manager.go:324] Recovery completed Nov 03 14:10:50.281062 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.281048 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:50.282766 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.282747 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:50.282863 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.282776 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:50.282863 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.282786 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:50.283248 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.283233 2566 cpu_manager.go:226] "Starting" policy="none" Nov 03 14:10:50.283307 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.283247 2566 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 14:10:50.283307 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.283266 2566 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 14:10:50.283307 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.283278 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:50.283887 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.283873 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:50.283939 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.283900 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:50.283939 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.283912 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:50.284476 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.284464 2566 policy_none.go:50] "Start" Nov 03 14:10:50.284536 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.284483 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 14:10:50.284536 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.284501 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 14:10:50.285399 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.285386 2566 policy_none.go:44] "Start" Nov 03 14:10:50.295818 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.295799 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-191.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 14:10:50.295924 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.295809 2566 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 14:10:50.296616 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.295800 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-191.ec2.internal.18e48e55aece6013 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-191.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-191.ec2.internal,},FirstTimestamp:2026-11-03 14:10:50.256424979 +0000 UTC m=+0.314918260,LastTimestamp:2026-11-03 14:10:50.256424979 +0000 UTC m=+0.314918260,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-191.ec2.internal,}" Nov 03 14:10:50.310792 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.310769 2566 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 14:10:50.310792 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.310770 2566 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 14:10:50.311036 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.310973 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-191.ec2.internal.18e48e55b06043da default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-191.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-191.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-191.ec2.internal,},FirstTimestamp:2026-11-03 14:10:50.282763226 +0000 UTC m=+0.341256507,LastTimestamp:2026-11-03 14:10:50.282763226 +0000 UTC m=+0.341256507,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-191.ec2.internal,}" Nov 03 14:10:50.311036 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.311030 2566 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 14:10:50.315301 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.315285 2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 14:10:50.317301 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.317286 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rprfq" Nov 03 14:10:50.319396 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.319372 2566 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 14:10:50.321700 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.321684 2566 manager.go:341] "Starting Device Plugin manager" Nov 03 14:10:50.322173 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.322157 2566 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 14:10:50.322173 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.322172 2566 server.go:87] "Starting device plugin registration server" Nov 03 14:10:50.322383 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.322369 2566 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 14:10:50.322463 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.322382 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 14:10:50.322514 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.322496 2566 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 14:10:50.322581 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.322570 2566 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 14:10:50.322581 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.322581 2566 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 14:10:50.323048 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.323028 2566 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 14:10:50.323126 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.323083 2566 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:50.327199 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.327183 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rprfq" Nov 03 14:10:50.338101 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.338041 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-191.ec2.internal.18e48e55b06089c1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-191.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-191.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-191.ec2.internal,},FirstTimestamp:2026-11-03 14:10:50.282781121 +0000 UTC m=+0.341274402,LastTimestamp:2026-11-03 14:10:50.282781121 +0000 UTC m=+0.341274402,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-191.ec2.internal,}" Nov 03 14:10:50.395108 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.395073 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 14:10:50.395108 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.395102 2566 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 14:10:50.396246 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.395122 2566 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 14:10:50.396246 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.395157 2566 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 14:10:50.412673 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.412652 2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:10:50.422727 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.422712 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:50.423530 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.423518 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:50.423593 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.423547 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:50.423593 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.423557 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:50.423593 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.423581 2566 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.442626 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.442588 2566 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.442626 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.442618 2566 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-191.ec2.internal\": node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:50.495580 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.495550 2566 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal"] Nov 03 14:10:50.495683 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.495609 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:50.496442 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.496424 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:50.496572 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.496457 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:50.496572 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.496473 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:50.496726 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.496711 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:50.498266 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.498252 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:50.498402 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.498390 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.498449 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.498418 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:50.499723 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.499706 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:50.499723 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.499737 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:50.499723 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.499753 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:50.499999 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.499774 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:50.499999 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.499797 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:50.499999 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.499811 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:50.500916 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.500901 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.501015 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.500926 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:50.501698 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.501684 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:50.501765 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.501712 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:50.501765 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.501727 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:50.516988 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.516964 2566 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-191.ec2.internal\" not found" node="ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.520508 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.520491 2566 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-191.ec2.internal\" not found" node="ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.566256 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.566230 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c5016cec399e6887c94ab5ea8dbdf29-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" (UID: \"3c5016cec399e6887c94ab5ea8dbdf29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.566371 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.566261 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c5016cec399e6887c94ab5ea8dbdf29-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" (UID: \"3c5016cec399e6887c94ab5ea8dbdf29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.566371 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.566286 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b01771a71144616672fedcb36938a552-config\") pod \"kube-apiserver-proxy-ip-10-0-130-191.ec2.internal\" (UID: \"b01771a71144616672fedcb36938a552\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.597454 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.597433 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:50.667148 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.667121 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c5016cec399e6887c94ab5ea8dbdf29-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" (UID: \"3c5016cec399e6887c94ab5ea8dbdf29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.667279 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.667156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c5016cec399e6887c94ab5ea8dbdf29-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" (UID: \"3c5016cec399e6887c94ab5ea8dbdf29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.667279 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.667180 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b01771a71144616672fedcb36938a552-config\") pod \"kube-apiserver-proxy-ip-10-0-130-191.ec2.internal\" (UID: \"b01771a71144616672fedcb36938a552\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.667279 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.667215 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c5016cec399e6887c94ab5ea8dbdf29-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" (UID: \"3c5016cec399e6887c94ab5ea8dbdf29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.667279 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.667246 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c5016cec399e6887c94ab5ea8dbdf29-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" (UID: \"3c5016cec399e6887c94ab5ea8dbdf29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.667279 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.667247 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b01771a71144616672fedcb36938a552-config\") pod \"kube-apiserver-proxy-ip-10-0-130-191.ec2.internal\" (UID: \"b01771a71144616672fedcb36938a552\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.698490 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.698426 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:50.798522 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.798495 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:50.820310 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.820285 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.824298 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:50.824275 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:10:50.899419 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.899240 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:50.999792 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:50.999708 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:51.100211 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:51.100180 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:51.200744 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:51.200715 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:51.206892 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.206867 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 14:10:51.207034 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.207016 2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 14:10:51.301441 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:51.301383 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:51.331128 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.331096 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 14:05:51 +0000 UTC" deadline="2028-07-13 04:07:39.241059714 +0000 UTC" Nov 03 14:10:51.331128 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.331126 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14821h56m47.909937026s" Nov 03 14:10:51.337934 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:51.337907 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01771a71144616672fedcb36938a552.slice/crio-999b1867c7515b06d1e6db3f17fe4b82cd262b99539615faabe4170a1711fd9b WatchSource:0}: Error finding container 999b1867c7515b06d1e6db3f17fe4b82cd262b99539615faabe4170a1711fd9b: Status 404 returned error can't find the container with id 999b1867c7515b06d1e6db3f17fe4b82cd262b99539615faabe4170a1711fd9b Nov 03 14:10:51.338140 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:51.338121 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5016cec399e6887c94ab5ea8dbdf29.slice/crio-69fc2d0a5d65d549a0169817881ddf3a96f96b3096dadd736528ef36dfbfe900 WatchSource:0}: Error finding container 69fc2d0a5d65d549a0169817881ddf3a96f96b3096dadd736528ef36dfbfe900: Status 404 returned error can't find the container with id 69fc2d0a5d65d549a0169817881ddf3a96f96b3096dadd736528ef36dfbfe900 Nov 03 14:10:51.341959 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.341939 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:10:51.399587 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.399527 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" event={"ID":"3c5016cec399e6887c94ab5ea8dbdf29","Type":"ContainerStarted","Data":"69fc2d0a5d65d549a0169817881ddf3a96f96b3096dadd736528ef36dfbfe900"} Nov 03 14:10:51.400600 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.400575 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" event={"ID":"b01771a71144616672fedcb36938a552","Type":"ContainerStarted","Data":"999b1867c7515b06d1e6db3f17fe4b82cd262b99539615faabe4170a1711fd9b"} Nov 03 14:10:51.401726 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:51.401711 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:51.476207 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.476178 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:51.477110 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.477095 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:51.477199 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.477124 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:51.477199 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.477136 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:51.502081 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:51.502063 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:51.506486 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.506462 2566 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:10:51.602979 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:51.602892 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:10:51.671024 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.670998 2566 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:10:51.764151 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.764112 2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:10:51.766356 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.766328 2566 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:10:51.791506 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.791469 2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 14:10:51.792488 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.792461 2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:10:51.812379 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:51.812353 2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 14:10:52.025403 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.025328 2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:10:52.244758 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.244732 2566 apiserver.go:52] "Watching apiserver" Nov 03 14:10:52.263907 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.263884 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 14:10:52.281736 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.281663 2566 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 14:10:52.282100 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.282075 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-tjsp9","kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal","openshift-dns/node-resolver-qnhkx","openshift-network-diagnostics/network-check-target-phmgc","openshift-network-operator/iptables-alerter-9pdj9","openshift-ovn-kubernetes/ovnkube-node-7j9ks","kube-system/konnectivity-agent-frkjs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs","openshift-image-registry/node-ca-dlwbh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal","openshift-multus/multus-additional-cni-plugins-99kd6","openshift-multus/multus-vlblr","openshift-multus/network-metrics-daemon-r5lc9","openshift-cluster-node-tuning-operator/tuned-747sw"] Nov 03 14:10:52.284356 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.284330 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:10:52.286630 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.286611 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:10:52.286730 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:52.286694 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:10:52.288391 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.288365 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:10:52.290378 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.290361 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:10:52.290469 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:52.290426 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:10:52.293173 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.292964 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:10:52.294630 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.294610 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.296761 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.296732 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:10:52.299371 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.299355 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:52.299464 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:52.299421 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:10:52.301968 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.301948 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 14:10:52.302057 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.302024 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.302129 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.302112 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lqmd6\"" Nov 03 14:10:52.302217 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.302205 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 14:10:52.304064 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.304009 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:52.307071 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.307051 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:52.309253 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.309232 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.316034 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.316013 2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 14:10:52.321126 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.321103 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t2fhf" Nov 03 14:10:52.330955 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.330799 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t2fhf" Nov 03 14:10:52.365302 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.365281 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 14:10:52.375942 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.375911 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:52.376056 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.375954 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkjnc\" (UniqueName: \"kubernetes.io/projected/05a67278-34b4-462d-b4cb-9250800bf3db-kube-api-access-wkjnc\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:52.376056 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.375981 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-systemd-units\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.376173 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376060 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.376173 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376108 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovnkube-config\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.376173 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376135 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-run\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.376173 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376167 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlv2\" (UniqueName: \"kubernetes.io/projected/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-kube-api-access-5dlv2\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.376354 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376188 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-system-cni-dir\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:52.376354 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376218 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-hostroot\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.376354 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376276 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-systemd\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.376354 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376331 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dch9b\" (UniqueName: \"kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b\") pod \"network-check-target-phmgc\" (UID: \"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf\") " pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:10:52.376535 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376371 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysctl-d\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.376535 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376412 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-multus-certs\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.376535 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376437 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-var-lib-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.376535 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376459 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-ovn\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.376535 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376507 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-tuned\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.376535 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376535 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/905b521a-2b6a-4fb6-ba5b-33311a2613b0-agent-certs\") pod \"konnectivity-agent-frkjs\" (UID: \"905b521a-2b6a-4fb6-ba5b-33311a2613b0\") " pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:10:52.376752 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376571 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-kubelet-config\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:52.376752 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376609 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:52.376752 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376651 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-etc-kubernetes\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.376752 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376684 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k45tm\" (UniqueName: \"kubernetes.io/projected/dad8ff83-33fe-4482-927e-8b70b1e72643-kube-api-access-k45tm\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:10:52.376752 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376714 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d4lt\" (UniqueName: \"kubernetes.io/projected/b586d272-c9c0-455f-aec9-333ae6d0c69a-kube-api-access-7d4lt\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:10:52.376752 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376744 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.376995 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376767 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-cni-netd\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.376995 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376783 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k6zn\" (UniqueName: \"kubernetes.io/projected/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-kube-api-access-4k6zn\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.376995 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376825 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-modprobe-d\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.376995 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376868 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d153593a-c25f-4bef-acca-ea7e50b32c87-cni-binary-copy\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.376995 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376900 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-cni-multus\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.376995 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376933 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-daemon-config\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.376995 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.376965 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:10:52.377236 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377001 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b586d272-c9c0-455f-aec9-333ae6d0c69a-tmp-dir\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:10:52.377236 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377048 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkjjv\" (UniqueName: \"kubernetes.io/projected/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-kube-api-access-xkjjv\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:10:52.377236 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377087 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-kubelet\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.377236 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377116 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq4pb\" (UniqueName: \"kubernetes.io/projected/40e20104-e97c-41b2-8530-abb2dec148a2-kube-api-access-kq4pb\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:52.377236 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377142 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-cni-binary-copy\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:52.377236 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377190 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-cni-bin\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.377476 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377251 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysctl-conf\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.377476 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377317 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-tmp\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.377476 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377363 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-conf-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.377476 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377418 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-env-overrides\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.377476 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377465 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-var-lib-kubelet\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.377686 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377494 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edb1d4a1-0656-4417-8b7d-86a68db196ee-host\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:10:52.377686 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377509 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-dbus\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:52.377686 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377541 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-device-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:52.377686 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377599 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-slash\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.377686 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377635 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-log-socket\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.377908 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377687 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-lib-modules\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.377908 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377753 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:52.377908 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377792 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-socket-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:52.377908 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377838 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:52.377908 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377892 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-system-cni-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.378122 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377920 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-kubelet\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.378122 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377934 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-run-netns\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.378122 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377954 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-etc-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.378122 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.377974 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-node-log\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.378122 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378001 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:52.378122 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378027 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-cni-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.378122 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378042 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-k8s-cni-cncf-io\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.378122 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378058 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-host-slash\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:10:52.378122 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378103 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysconfig\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.378446 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378127 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-sys\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.378446 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378148 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-registration-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:52.378446 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378174 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-iptables-alerter-script\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:10:52.378446 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378197 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.378446 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378219 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-systemd\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.378446 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378242 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edb1d4a1-0656-4417-8b7d-86a68db196ee-serviceca\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:10:52.378446 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378266 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:52.378446 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378292 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-cnibin\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.378446 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378313 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-cni-bin\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.378446 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378338 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mw4d\" (UniqueName: \"kubernetes.io/projected/edb1d4a1-0656-4417-8b7d-86a68db196ee-kube-api-access-8mw4d\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:10:52.378446 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378362 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-socket-dir-parent\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.378809 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378444 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-netns\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.378809 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378467 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b586d272-c9c0-455f-aec9-333ae6d0c69a-hosts-file\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:10:52.378809 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378481 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovnkube-script-lib\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.378809 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378504 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-kubernetes\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.378809 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378521 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/905b521a-2b6a-4fb6-ba5b-33311a2613b0-konnectivity-ca\") pod \"konnectivity-agent-frkjs\" (UID: \"905b521a-2b6a-4fb6-ba5b-33311a2613b0\") " pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:10:52.378809 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378541 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-cnibin\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:52.378809 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378558 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2mvt\" (UniqueName: \"kubernetes.io/projected/d153593a-c25f-4bef-acca-ea7e50b32c87-kube-api-access-z2mvt\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:52.378809 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378578 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovn-node-metrics-cert\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:52.378809 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378594 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-host\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:52.378809 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.378611 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-sys-fs\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:52.403306 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403271 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 14:10:52.403476 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403405 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 14:10:52.403550 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403536 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-95wl7\"" Nov 03 14:10:52.403606 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403555 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 14:10:52.403606 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403583 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 14:10:52.403702 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403587 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 14:10:52.403702 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403648 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trbj6\"" Nov 03 14:10:52.403702 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403655 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 14:10:52.403866 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403703 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 14:10:52.403866 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403723 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2ft6v\"" Nov 03 14:10:52.403866 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403731 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 14:10:52.403866 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403751 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 14:10:52.403866 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403761 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 14:10:52.403866 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403766 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xm8pf\"" Nov 03 14:10:52.403866 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403772 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 14:10:52.403866 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403813 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qgxkz\"" Nov 03 14:10:52.403866 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403755 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 14:10:52.403866 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403810 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:10:52.403866 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403780 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403830 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403886 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403859 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403906 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403916 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403933 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qhrll\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403942 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403975 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.403999 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.404006 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tb44c\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.404030 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.404045 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.404054 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q495v\"" Nov 03 14:10:52.404250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:52.404151 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 14:10:53.752992 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.752960 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-env-overrides\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.752992 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753002 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-var-lib-kubelet\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.753234 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753027 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edb1d4a1-0656-4417-8b7d-86a68db196ee-host\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:10:53.753234 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753054 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-dbus\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:53.753234 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753091 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-device-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.753234 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753117 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-slash\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.753234 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753117 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-var-lib-kubelet\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.753234 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753144 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-log-socket\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.753234 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753167 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-lib-modules\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.753234 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.753234 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753213 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-socket-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753246 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753271 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-system-cni-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753293 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-kubelet\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753313 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-run-netns\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753328 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-etc-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753333 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-dbus\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753343 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-node-log\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753371 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753380 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edb1d4a1-0656-4417-8b7d-86a68db196ee-host\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753387 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-cni-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753448 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-cni-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753488 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-system-cni-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.753508 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753513 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-kubelet\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.753914 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753539 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-run-netns\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.753914 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753564 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-etc-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.753914 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753587 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-node-log\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.753914 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753769 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.753914 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753907 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-lib-modules\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.754060 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.753994 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-socket-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.754060 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754030 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-device-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.754126 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754065 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-slash\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.754259 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754245 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-k8s-cni-cncf-io\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.754378 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754360 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-log-socket\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.754487 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754464 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-env-overrides\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.754630 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754162 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-k8s-cni-cncf-io\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.754630 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754589 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-host-slash\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:10:53.754740 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754636 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysconfig\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.754740 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754662 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-sys\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.754740 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754666 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-host-slash\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:10:53.754740 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754685 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-registration-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.754740 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754735 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysconfig\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754744 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-sys\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754735 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754755 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754758 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-iptables-alerter-script\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754783 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-registration-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754790 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754819 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-systemd\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754825 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754847 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edb1d4a1-0656-4417-8b7d-86a68db196ee-serviceca\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754871 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754874 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-systemd\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754893 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-cnibin\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754915 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-cni-bin\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.754951 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754938 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mw4d\" (UniqueName: \"kubernetes.io/projected/edb1d4a1-0656-4417-8b7d-86a68db196ee-kube-api-access-8mw4d\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754965 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-socket-dir-parent\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.754989 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-netns\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:53.754997 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755009 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b586d272-c9c0-455f-aec9-333ae6d0c69a-hosts-file\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755033 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovnkube-script-lib\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:53.755086 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret podName:fc214996-6ee2-4bf3-a2cd-49cd4eceaa09 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:54.255042674 +0000 UTC m=+3.039771910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret") pod "global-pull-secret-syncer-tjsp9" (UID: "fc214996-6ee2-4bf3-a2cd-49cd4eceaa09") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755107 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-kubernetes\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755138 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/905b521a-2b6a-4fb6-ba5b-33311a2613b0-konnectivity-ca\") pod \"konnectivity-agent-frkjs\" (UID: \"905b521a-2b6a-4fb6-ba5b-33311a2613b0\") " pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755164 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-cnibin\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755189 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2mvt\" (UniqueName: \"kubernetes.io/projected/d153593a-c25f-4bef-acca-ea7e50b32c87-kube-api-access-z2mvt\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755215 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovn-node-metrics-cert\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755237 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-host\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755252 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-sys-fs\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755266 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkjnc\" (UniqueName: \"kubernetes.io/projected/05a67278-34b4-462d-b4cb-9250800bf3db-kube-api-access-wkjnc\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755287 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edb1d4a1-0656-4417-8b7d-86a68db196ee-serviceca\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:10:53.755482 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755295 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-systemd-units\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755310 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755325 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovnkube-config\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755340 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-run\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlv2\" (UniqueName: \"kubernetes.io/projected/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-kube-api-access-5dlv2\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755359 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-iptables-alerter-script\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755387 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-system-cni-dir\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755410 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-hostroot\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-systemd\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755464 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dch9b\" (UniqueName: \"kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b\") pod \"network-check-target-phmgc\" (UID: \"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf\") " pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755491 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysctl-d\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755489 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovnkube-script-lib\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755540 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-socket-dir-parent\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755554 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-kubernetes\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755573 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-multus-certs\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755601 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-var-lib-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755599 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.756133 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755648 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-ovn\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755668 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-cnibin\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755670 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-tuned\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755712 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/905b521a-2b6a-4fb6-ba5b-33311a2613b0-agent-certs\") pod \"konnectivity-agent-frkjs\" (UID: \"905b521a-2b6a-4fb6-ba5b-33311a2613b0\") " pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755738 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-kubelet-config\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755763 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755787 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-etc-kubernetes\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755811 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k45tm\" (UniqueName: \"kubernetes.io/projected/dad8ff83-33fe-4482-927e-8b70b1e72643-kube-api-access-k45tm\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755827 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d4lt\" (UniqueName: \"kubernetes.io/projected/b586d272-c9c0-455f-aec9-333ae6d0c69a-kube-api-access-7d4lt\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755846 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-systemd-units\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755851 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755882 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755884 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-cni-bin\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755936 2566 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756075 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-var-lib-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756124 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-hostroot\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755429 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-sys-fs\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756159 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-systemd\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.756969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756441 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-etc-kubernetes\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756447 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-kubelet-config\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756504 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-netns\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756510 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b586d272-c9c0-455f-aec9-333ae6d0c69a-hosts-file\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756544 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-cni-netd\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756554 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-ovn\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756574 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k6zn\" (UniqueName: \"kubernetes.io/projected/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-kube-api-access-4k6zn\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756132 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-system-cni-dir\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756585 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysctl-d\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756592 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-modprobe-d\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756646 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d153593a-c25f-4bef-acca-ea7e50b32c87-cni-binary-copy\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756650 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-multus-certs\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756676 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-cni-multus\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756705 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-daemon-config\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756711 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-modprobe-d\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756755 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756766 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756797 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-cni-netd\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.757770 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756830 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-cni-multus\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:53.756839 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756866 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-host\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:53.756895 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs podName:dad8ff83-33fe-4482-927e-8b70b1e72643 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:54.256879215 +0000 UTC m=+3.041608444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs") pod "network-metrics-daemon-r5lc9" (UID: "dad8ff83-33fe-4482-927e-8b70b1e72643") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.755941 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-run\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756909 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b586d272-c9c0-455f-aec9-333ae6d0c69a-tmp-dir\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.756969 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-cnibin\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757005 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjjv\" (UniqueName: \"kubernetes.io/projected/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-kube-api-access-xkjjv\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757043 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-kubelet\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757067 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq4pb\" (UniqueName: \"kubernetes.io/projected/40e20104-e97c-41b2-8530-abb2dec148a2-kube-api-access-kq4pb\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757109 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-cni-binary-copy\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757132 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/905b521a-2b6a-4fb6-ba5b-33311a2613b0-konnectivity-ca\") pod \"konnectivity-agent-frkjs\" (UID: \"905b521a-2b6a-4fb6-ba5b-33311a2613b0\") " pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757135 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-cni-bin\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757163 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-cni-bin\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757189 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysctl-conf\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757213 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-tmp\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757237 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-conf-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.758557 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757111 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.759116 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757240 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b586d272-c9c0-455f-aec9-333ae6d0c69a-tmp-dir\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:10:53.759116 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757813 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovnkube-config\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.759116 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757880 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-kubelet\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.759116 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757896 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-daemon-config\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.759116 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.757959 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-cni-binary-copy\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.759116 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.758013 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-conf-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.759116 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.758030 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysctl-conf\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.759116 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.758458 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d153593a-c25f-4bef-acca-ea7e50b32c87-cni-binary-copy\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.760023 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.759998 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovn-node-metrics-cert\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.760671 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.760647 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/905b521a-2b6a-4fb6-ba5b-33311a2613b0-agent-certs\") pod \"konnectivity-agent-frkjs\" (UID: \"905b521a-2b6a-4fb6-ba5b-33311a2613b0\") " pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:10:53.762458 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.762436 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-tmp\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.769395 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.769374 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-tuned\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.780311 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:53.780286 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:10:53.780311 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:53.780315 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:10:53.780585 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:53.780329 2566 projected.go:196] Error preparing data for projected volume kube-api-access-dch9b for pod openshift-network-diagnostics/network-check-target-phmgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:53.780585 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:53.780391 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b podName:6f1c105f-a7fa-4aeb-b8de-a413f5def9bf nodeName:}" failed. No retries permitted until 2026-11-03 14:10:54.280372614 +0000 UTC m=+3.065101851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dch9b" (UniqueName: "kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b") pod "network-check-target-phmgc" (UID: "6f1c105f-a7fa-4aeb-b8de-a413f5def9bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:53.782941 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.782914 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkjnc\" (UniqueName: \"kubernetes.io/projected/05a67278-34b4-462d-b4cb-9250800bf3db-kube-api-access-wkjnc\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.784682 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.784658 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d4lt\" (UniqueName: \"kubernetes.io/projected/b586d272-c9c0-455f-aec9-333ae6d0c69a-kube-api-access-7d4lt\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:10:53.784773 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.784737 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k45tm\" (UniqueName: \"kubernetes.io/projected/dad8ff83-33fe-4482-927e-8b70b1e72643-kube-api-access-k45tm\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:10:53.785128 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.785106 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlv2\" (UniqueName: \"kubernetes.io/projected/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-kube-api-access-5dlv2\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.785530 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.785507 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2mvt\" (UniqueName: \"kubernetes.io/projected/d153593a-c25f-4bef-acca-ea7e50b32c87-kube-api-access-z2mvt\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.785644 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.785531 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mw4d\" (UniqueName: \"kubernetes.io/projected/edb1d4a1-0656-4417-8b7d-86a68db196ee-kube-api-access-8mw4d\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:10:53.785969 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.785947 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq4pb\" (UniqueName: \"kubernetes.io/projected/40e20104-e97c-41b2-8530-abb2dec148a2-kube-api-access-kq4pb\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.786051 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.785947 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjjv\" (UniqueName: \"kubernetes.io/projected/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-kube-api-access-xkjjv\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:10:53.787172 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.787150 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k6zn\" (UniqueName: \"kubernetes.io/projected/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-kube-api-access-4k6zn\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.872924 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.872699 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:10:53.880574 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.880551 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:10:53.886492 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:53.886453 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905b521a_2b6a_4fb6_ba5b_33311a2613b0.slice/crio-695791e9fed3c00fe43512d295793f792f61086f14c4ae256b88615f2c24e9b9 WatchSource:0}: Error finding container 695791e9fed3c00fe43512d295793f792f61086f14c4ae256b88615f2c24e9b9: Status 404 returned error can't find the container with id 695791e9fed3c00fe43512d295793f792f61086f14c4ae256b88615f2c24e9b9 Nov 03 14:10:53.891473 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:53.891453 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb586d272_c9c0_455f_aec9_333ae6d0c69a.slice/crio-4083a692b9eb0f2a859ddcc5f55cd9a3aff7c5aaa4d334def8e0f2df046c5530 WatchSource:0}: Error finding container 4083a692b9eb0f2a859ddcc5f55cd9a3aff7c5aaa4d334def8e0f2df046c5530: Status 404 returned error can't find the container with id 4083a692b9eb0f2a859ddcc5f55cd9a3aff7c5aaa4d334def8e0f2df046c5530 Nov 03 14:10:53.892971 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.892767 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:10:53.902401 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:53.902379 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b9aee7_f7fb_4a15_8527_9ad5bda2523d.slice/crio-f0d7a728bd64603d00f7123064da977fb3e58c376216ddea8902aa82ba66a3b0 WatchSource:0}: Error finding container f0d7a728bd64603d00f7123064da977fb3e58c376216ddea8902aa82ba66a3b0: Status 404 returned error can't find the container with id f0d7a728bd64603d00f7123064da977fb3e58c376216ddea8902aa82ba66a3b0 Nov 03 14:10:53.912986 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.912961 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:10:53.918083 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.917963 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:10:53.920696 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.920674 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:10:53.928095 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.928077 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:10:53.931226 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:53.931199 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a85509_0811_4aaf_a86d_b41e5ae5a9cc.slice/crio-6f2ab086794d480503a53b41b8a854f866fe036b494e8644cc5bf586e1cdbb37 WatchSource:0}: Error finding container 6f2ab086794d480503a53b41b8a854f866fe036b494e8644cc5bf586e1cdbb37: Status 404 returned error can't find the container with id 6f2ab086794d480503a53b41b8a854f866fe036b494e8644cc5bf586e1cdbb37 Nov 03 14:10:53.937566 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:53.937538 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00819ff_6b3d_4e36_995b_2d671e0dc9f6.slice/crio-dd5812a2cd2c14e4f4b49defad789b0428f4e5646bc84c4fc489a413dd085632 WatchSource:0}: Error finding container dd5812a2cd2c14e4f4b49defad789b0428f4e5646bc84c4fc489a413dd085632: Status 404 returned error can't find the container with id dd5812a2cd2c14e4f4b49defad789b0428f4e5646bc84c4fc489a413dd085632 Nov 03 14:10:53.940029 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:53.940002 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e20104_e97c_41b2_8530_abb2dec148a2.slice/crio-c0b128e2f7a5c2e0f1faac54a5a1a33b7aab3700813da44785a9d1d0fe406f2e WatchSource:0}: Error finding container c0b128e2f7a5c2e0f1faac54a5a1a33b7aab3700813da44785a9d1d0fe406f2e: Status 404 returned error can't find the container with id c0b128e2f7a5c2e0f1faac54a5a1a33b7aab3700813da44785a9d1d0fe406f2e Nov 03 14:10:53.943447 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.943423 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vlblr" Nov 03 14:10:53.946962 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:53.946761 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:10:53.954248 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:10:53.954092 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd153593a_c25f_4bef_acca_ea7e50b32c87.slice/crio-2b32bff675427e2f8532946e3b2d9082bd1900a0c52210171e9a90ec60c5265b WatchSource:0}: Error finding container 2b32bff675427e2f8532946e3b2d9082bd1900a0c52210171e9a90ec60c5265b: Status 404 returned error can't find the container with id 2b32bff675427e2f8532946e3b2d9082bd1900a0c52210171e9a90ec60c5265b Nov 03 14:10:54.260054 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.259781 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:54.260054 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.259870 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:10:54.260054 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:54.259899 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:54.260054 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:54.259965 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret podName:fc214996-6ee2-4bf3-a2cd-49cd4eceaa09 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:55.259946022 +0000 UTC m=+4.044675254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret") pod "global-pull-secret-syncer-tjsp9" (UID: "fc214996-6ee2-4bf3-a2cd-49cd4eceaa09") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:54.260054 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:54.259991 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:54.260054 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:54.260030 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs podName:dad8ff83-33fe-4482-927e-8b70b1e72643 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:55.260018311 +0000 UTC m=+4.044747539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs") pod "network-metrics-daemon-r5lc9" (UID: "dad8ff83-33fe-4482-927e-8b70b1e72643") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:54.361850 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.361204 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dch9b\" (UniqueName: \"kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b\") pod \"network-check-target-phmgc\" (UID: \"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf\") " pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:10:54.361850 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:54.361355 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:10:54.361850 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:54.361372 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:10:54.361850 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:54.361389 2566 projected.go:196] Error preparing data for projected volume kube-api-access-dch9b for pod openshift-network-diagnostics/network-check-target-phmgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:54.361850 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:54.361448 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b podName:6f1c105f-a7fa-4aeb-b8de-a413f5def9bf nodeName:}" failed. No retries permitted until 2026-11-03 14:10:55.361431195 +0000 UTC m=+4.146160433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dch9b" (UniqueName: "kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b") pod "network-check-target-phmgc" (UID: "6f1c105f-a7fa-4aeb-b8de-a413f5def9bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:54.608357 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.608187 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 14:05:53 +0000 UTC" deadline="2028-07-12 09:50:22.536884494 +0000 UTC" Nov 03 14:10:54.608357 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.608248 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14803h39m27.92864035s" Nov 03 14:10:54.688679 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.688647 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" event={"ID":"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc","Type":"ContainerStarted","Data":"6f2ab086794d480503a53b41b8a854f866fe036b494e8644cc5bf586e1cdbb37"} Nov 03 14:10:54.695975 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.695939 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dlwbh" event={"ID":"edb1d4a1-0656-4417-8b7d-86a68db196ee","Type":"ContainerStarted","Data":"aff3c5989d7f586de304608ba8eaced771e012ed3f4770980b7aeb7b8d3fa9ee"} Nov 03 14:10:54.698269 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.698239 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9pdj9" event={"ID":"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d","Type":"ContainerStarted","Data":"f0d7a728bd64603d00f7123064da977fb3e58c376216ddea8902aa82ba66a3b0"} Nov 03 14:10:54.707329 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.707246 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qnhkx" event={"ID":"b586d272-c9c0-455f-aec9-333ae6d0c69a","Type":"ContainerStarted","Data":"4083a692b9eb0f2a859ddcc5f55cd9a3aff7c5aaa4d334def8e0f2df046c5530"} Nov 03 14:10:54.726841 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.726804 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" event={"ID":"b01771a71144616672fedcb36938a552","Type":"ContainerStarted","Data":"7ac05325799f5f60816b433404b7b4118942f1daecdadb85c75eb7196e30bbfb"} Nov 03 14:10:54.737494 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.737461 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vlblr" event={"ID":"d153593a-c25f-4bef-acca-ea7e50b32c87","Type":"ContainerStarted","Data":"2b32bff675427e2f8532946e3b2d9082bd1900a0c52210171e9a90ec60c5265b"} Nov 03 14:10:54.742646 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.742555 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-frkjs" event={"ID":"905b521a-2b6a-4fb6-ba5b-33311a2613b0","Type":"ContainerStarted","Data":"695791e9fed3c00fe43512d295793f792f61086f14c4ae256b88615f2c24e9b9"} Nov 03 14:10:54.752061 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.752026 2566 generic.go:356] "Generic (PLEG): container finished" podID="3c5016cec399e6887c94ab5ea8dbdf29" containerID="1a8e0dbc8ce266e152d48a70ee62821ad8999e34386a76bbc3d292a321dad5ef" exitCode=0 Nov 03 14:10:54.752188 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.752100 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" event={"ID":"3c5016cec399e6887c94ab5ea8dbdf29","Type":"ContainerDied","Data":"1a8e0dbc8ce266e152d48a70ee62821ad8999e34386a76bbc3d292a321dad5ef"} Nov 03 14:10:54.759896 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.759806 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99kd6" event={"ID":"05a67278-34b4-462d-b4cb-9250800bf3db","Type":"ContainerStarted","Data":"d762ba31bc07b465c2882275574c371615a22b61191d8c4c8ff5bc614c715e6d"} Nov 03 14:10:54.769422 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.769341 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" event={"ID":"40e20104-e97c-41b2-8530-abb2dec148a2","Type":"ContainerStarted","Data":"c0b128e2f7a5c2e0f1faac54a5a1a33b7aab3700813da44785a9d1d0fe406f2e"} Nov 03 14:10:54.773982 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.773925 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" podStartSLOduration=1.7739088710000002 podStartE2EDuration="1.773908871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:54.773639689 +0000 UTC m=+3.558368925" watchObservedRunningTime="2026-11-03 14:10:54.773908871 +0000 UTC m=+3.558638110" Nov 03 14:10:54.775720 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:54.775694 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-747sw" event={"ID":"b00819ff-6b3d-4e36-995b-2d671e0dc9f6","Type":"ContainerStarted","Data":"dd5812a2cd2c14e4f4b49defad789b0428f4e5646bc84c4fc489a413dd085632"} Nov 03 14:10:55.270335 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:55.269460 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:10:55.270335 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:55.269525 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:55.270335 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:55.269692 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:55.270335 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:55.269750 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret podName:fc214996-6ee2-4bf3-a2cd-49cd4eceaa09 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:57.269733952 +0000 UTC m=+6.054463169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret") pod "global-pull-secret-syncer-tjsp9" (UID: "fc214996-6ee2-4bf3-a2cd-49cd4eceaa09") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:55.270335 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:55.270140 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:55.270335 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:55.270186 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs podName:dad8ff83-33fe-4482-927e-8b70b1e72643 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:57.270172366 +0000 UTC m=+6.054901595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs") pod "network-metrics-daemon-r5lc9" (UID: "dad8ff83-33fe-4482-927e-8b70b1e72643") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:55.370921 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:55.370853 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dch9b\" (UniqueName: \"kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b\") pod \"network-check-target-phmgc\" (UID: \"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf\") " pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:10:55.371104 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:55.371090 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:10:55.371179 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:55.371109 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:10:55.371179 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:55.371124 2566 projected.go:196] Error preparing data for projected volume kube-api-access-dch9b for pod openshift-network-diagnostics/network-check-target-phmgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:55.371280 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:55.371180 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b podName:6f1c105f-a7fa-4aeb-b8de-a413f5def9bf nodeName:}" failed. No retries permitted until 2026-11-03 14:10:57.371161479 +0000 UTC m=+6.155890711 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dch9b" (UniqueName: "kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b") pod "network-check-target-phmgc" (UID: "6f1c105f-a7fa-4aeb-b8de-a413f5def9bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:55.608833 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:55.608692 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 14:05:53 +0000 UTC" deadline="2028-08-14 23:07:31.91482896 +0000 UTC" Nov 03 14:10:55.608833 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:55.608749 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15608h56m36.306083693s" Nov 03 14:10:55.672227 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:55.669730 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:10:55.672227 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:55.669869 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:10:55.672227 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:55.670248 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:10:55.672227 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:55.670340 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:10:55.672227 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:55.670403 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:55.672227 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:55.670472 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:10:55.791217 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:55.791187 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" event={"ID":"3c5016cec399e6887c94ab5ea8dbdf29","Type":"ContainerStarted","Data":"5aaacc2082e87350c4c9aad261d28a42e57c300877933879c2dfe03e0d9800d7"} Nov 03 14:10:56.800170 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:56.799579 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99kd6" event={"ID":"05a67278-34b4-462d-b4cb-9250800bf3db","Type":"ContainerStarted","Data":"7cce28818517e048d93a205f99151f528c121695e1ccd1bdbeade5577d9a47d6"} Nov 03 14:10:56.879782 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:56.879699 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" podStartSLOduration=3.879678525 podStartE2EDuration="3.879678525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:55.832899841 +0000 UTC m=+4.617629077" watchObservedRunningTime="2026-11-03 14:10:56.879678525 +0000 UTC m=+5.664407765" Nov 03 14:10:57.291663 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:57.291603 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:10:57.291838 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:57.291688 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:57.291838 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:57.291810 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:57.291961 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:57.291870 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret podName:fc214996-6ee2-4bf3-a2cd-49cd4eceaa09 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:01.291854163 +0000 UTC m=+10.076583381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret") pod "global-pull-secret-syncer-tjsp9" (UID: "fc214996-6ee2-4bf3-a2cd-49cd4eceaa09") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:57.292253 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:57.292235 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:57.292335 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:57.292287 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs podName:dad8ff83-33fe-4482-927e-8b70b1e72643 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:01.292272692 +0000 UTC m=+10.077001922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs") pod "network-metrics-daemon-r5lc9" (UID: "dad8ff83-33fe-4482-927e-8b70b1e72643") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:57.392730 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:57.392696 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dch9b\" (UniqueName: \"kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b\") pod \"network-check-target-phmgc\" (UID: \"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf\") " pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:10:57.392944 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:57.392835 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:10:57.392944 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:57.392853 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:10:57.392944 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:57.392863 2566 projected.go:196] Error preparing data for projected volume kube-api-access-dch9b for pod openshift-network-diagnostics/network-check-target-phmgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:57.392944 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:57.392917 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b podName:6f1c105f-a7fa-4aeb-b8de-a413f5def9bf nodeName:}" failed. No retries permitted until 2026-11-03 14:11:01.392897913 +0000 UTC m=+10.177627148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dch9b" (UniqueName: "kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b") pod "network-check-target-phmgc" (UID: "6f1c105f-a7fa-4aeb-b8de-a413f5def9bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:57.671847 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:57.671237 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:10:57.671847 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:57.671401 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:10:57.671847 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:57.671705 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:10:57.671847 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:57.671804 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:10:57.672340 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:57.672204 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:57.672340 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:57.672302 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:10:57.807869 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:57.806957 2566 generic.go:356] "Generic (PLEG): container finished" podID="05a67278-34b4-462d-b4cb-9250800bf3db" containerID="7cce28818517e048d93a205f99151f528c121695e1ccd1bdbeade5577d9a47d6" exitCode=0 Nov 03 14:10:57.807869 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:57.807022 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99kd6" event={"ID":"05a67278-34b4-462d-b4cb-9250800bf3db","Type":"ContainerDied","Data":"7cce28818517e048d93a205f99151f528c121695e1ccd1bdbeade5577d9a47d6"} Nov 03 14:10:58.815639 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:58.815587 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dlwbh" event={"ID":"edb1d4a1-0656-4417-8b7d-86a68db196ee","Type":"ContainerStarted","Data":"d8bf1a621b000630cfc9445f57d2bcf16ddfa354709cbf69feafc679ef0a1035"} Nov 03 14:10:58.820319 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:58.820290 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-frkjs" event={"ID":"905b521a-2b6a-4fb6-ba5b-33311a2613b0","Type":"ContainerStarted","Data":"53d529611ab9df6e251ae00eb107bd0d9ed7c9d0ea411db99c31ff4672012c65"} Nov 03 14:10:58.858421 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:58.857410 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dlwbh" podStartSLOduration=3.208701395 podStartE2EDuration="7.857392743s" totalImagesPullingTime="4.648691348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:58.857251156 +0000 UTC m=+7.641980406" watchObservedRunningTime="2026-11-03 14:10:58.857392743 +0000 UTC m=+7.642121975" Nov 03 14:10:58.901305 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:58.901231 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-frkjs" podStartSLOduration=4.139284209 podStartE2EDuration="7.901216506s" totalImagesPullingTime="3.761932297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:58.900642932 +0000 UTC m=+7.685372172" watchObservedRunningTime="2026-11-03 14:10:58.901216506 +0000 UTC m=+7.685945743" Nov 03 14:10:59.669457 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:59.669426 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:10:59.669879 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:59.669567 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:10:59.669879 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:59.669720 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:10:59.669879 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:59.669831 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:10:59.670362 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:59.670239 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:10:59.670362 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:10:59.670327 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:10:59.826106 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:10:59.826012 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" event={"ID":"40e20104-e97c-41b2-8530-abb2dec148a2","Type":"ContainerStarted","Data":"78a3f54efe65d17c555640198c53c3ef64ce6df63752180b65f2ab1f8ff80dd8"} Nov 03 14:11:00.836065 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:00.832496 2566 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 14:11:00.842977 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:00.842945 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" event={"ID":"40e20104-e97c-41b2-8530-abb2dec148a2","Type":"ContainerStarted","Data":"e10ea597bcf290061cfc260a452eaa53af554b4f309909189590fd1d3cb65545"} Nov 03 14:11:01.328627 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:01.328573 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:01.328814 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:01.328655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:01.328814 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:01.328800 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:11:01.328912 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:01.328864 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret podName:fc214996-6ee2-4bf3-a2cd-49cd4eceaa09 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:09.328842703 +0000 UTC m=+18.113571937 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret") pod "global-pull-secret-syncer-tjsp9" (UID: "fc214996-6ee2-4bf3-a2cd-49cd4eceaa09") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:11:01.329308 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:01.329285 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:11:01.329429 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:01.329340 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs podName:dad8ff83-33fe-4482-927e-8b70b1e72643 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:09.329326103 +0000 UTC m=+18.114055320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs") pod "network-metrics-daemon-r5lc9" (UID: "dad8ff83-33fe-4482-927e-8b70b1e72643") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:11:01.429787 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:01.429751 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dch9b\" (UniqueName: \"kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b\") pod \"network-check-target-phmgc\" (UID: \"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf\") " pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:01.429957 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:01.429942 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:11:01.430000 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:01.429961 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:11:01.430000 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:01.429970 2566 projected.go:196] Error preparing data for projected volume kube-api-access-dch9b for pod openshift-network-diagnostics/network-check-target-phmgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:11:01.430062 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:01.430014 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b podName:6f1c105f-a7fa-4aeb-b8de-a413f5def9bf nodeName:}" failed. No retries permitted until 2026-11-03 14:11:09.430003118 +0000 UTC m=+18.214732346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dch9b" (UniqueName: "kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b") pod "network-check-target-phmgc" (UID: "6f1c105f-a7fa-4aeb-b8de-a413f5def9bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:11:01.601034 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:01.600942 2566 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T14:11:00.832524035Z","UUID":"c9776ae2-40e9-44c3-bc86-074e9177fa92","Handler":null,"Name":"","Endpoint":""} Nov 03 14:11:01.602935 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:01.602915 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 14:11:01.603055 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:01.602943 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 14:11:01.671412 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:01.669719 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:01.671412 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:01.669865 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:01.671412 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:01.671275 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:01.671412 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:01.671358 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:01.672022 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:01.671839 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:01.672022 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:01.671935 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:01.849466 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:01.849429 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vlblr" event={"ID":"d153593a-c25f-4bef-acca-ea7e50b32c87","Type":"ContainerStarted","Data":"834d34f3701373b8bdb6fe0358873f8a9144731d07f0e96d26c678a58dea7621"} Nov 03 14:11:02.853414 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:02.853381 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qnhkx" event={"ID":"b586d272-c9c0-455f-aec9-333ae6d0c69a","Type":"ContainerStarted","Data":"7876c6bb212070e7f66b5fa7728225411f260f8c5c08b882818542bbb6fc29bf"} Nov 03 14:11:02.855705 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:02.855675 2566 generic.go:356] "Generic (PLEG): container finished" podID="05a67278-34b4-462d-b4cb-9250800bf3db" containerID="e9cde02f06b5a90d447e5c502d0fbc4efa76d72f604f4bb7c78337db3f12fcb4" exitCode=0 Nov 03 14:11:02.855853 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:02.855779 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99kd6" event={"ID":"05a67278-34b4-462d-b4cb-9250800bf3db","Type":"ContainerDied","Data":"e9cde02f06b5a90d447e5c502d0fbc4efa76d72f604f4bb7c78337db3f12fcb4"} Nov 03 14:11:02.862214 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:02.862181 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" event={"ID":"40e20104-e97c-41b2-8530-abb2dec148a2","Type":"ContainerStarted","Data":"29edfbb956046e958828b1020630754fb31293e7c262a57ebe1c1718219fdac3"} Nov 03 14:11:02.896372 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:02.896320 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vlblr" podStartSLOduration=4.341721717 podStartE2EDuration="11.89630388s" totalImagesPullingTime="7.554582163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:01.896500141 +0000 UTC m=+10.681229379" watchObservedRunningTime="2026-11-03 14:11:02.89630388 +0000 UTC m=+11.681033118" Nov 03 14:11:02.896675 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:02.896641 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qnhkx" podStartSLOduration=4.107663871 podStartE2EDuration="11.896632603s" totalImagesPullingTime="7.788968732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:02.896031711 +0000 UTC m=+11.680760949" watchObservedRunningTime="2026-11-03 14:11:02.896632603 +0000 UTC m=+11.681361840" Nov 03 14:11:02.987910 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:02.987849 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" podStartSLOduration=4.394423075 podStartE2EDuration="11.987829935s" totalImagesPullingTime="7.59340686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:02.987209918 +0000 UTC m=+11.771939151" watchObservedRunningTime="2026-11-03 14:11:02.987829935 +0000 UTC m=+11.772559173" Nov 03 14:11:03.670204 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:03.669965 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:03.670407 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:03.670041 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:03.670407 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:03.670310 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:03.670487 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:03.670411 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:03.670487 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:03.670091 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:03.670575 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:03.670529 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:03.780793 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:03.780763 2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:11:03.782112 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:03.782089 2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:11:03.867096 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:03.867055 2566 generic.go:356] "Generic (PLEG): container finished" podID="05a67278-34b4-462d-b4cb-9250800bf3db" containerID="963aabbb5fe1c30f57377f07047a35d49dad86b249a6046ffe2bf2c7a97e7397" exitCode=0 Nov 03 14:11:03.867664 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:03.867143 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99kd6" event={"ID":"05a67278-34b4-462d-b4cb-9250800bf3db","Type":"ContainerDied","Data":"963aabbb5fe1c30f57377f07047a35d49dad86b249a6046ffe2bf2c7a97e7397"} Nov 03 14:11:03.870027 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:03.869939 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-747sw" event={"ID":"b00819ff-6b3d-4e36-995b-2d671e0dc9f6","Type":"ContainerStarted","Data":"e3692f7aa6801967e2ee395ebaebaae4699a02e0c5002926c800903807c53566"} Nov 03 14:11:03.872428 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:03.871892 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9pdj9" event={"ID":"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d","Type":"ContainerStarted","Data":"08cc1c52fe08ce3d17eb78abd0fb6b9ea6583f7b897a24e23301777ae71db5eb"} Nov 03 14:11:03.964844 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:03.964797 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9pdj9" podStartSLOduration=5.288599687 podStartE2EDuration="12.964783791s" totalImagesPullingTime="7.676184104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:03.964713996 +0000 UTC m=+12.749443234" watchObservedRunningTime="2026-11-03 14:11:03.964783791 +0000 UTC m=+12.749513028" Nov 03 14:11:04.008754 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:04.008703 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-747sw" podStartSLOduration=3.166814702 podStartE2EDuration="13.00868881s" totalImagesPullingTime="9.841874108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:04.008406699 +0000 UTC m=+12.793135938" watchObservedRunningTime="2026-11-03 14:11:04.00868881 +0000 UTC m=+12.793418093" Nov 03 14:11:05.512760 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:05.512729 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qnhkx_b586d272-c9c0-455f-aec9-333ae6d0c69a/dns-node-resolver/0.log" Nov 03 14:11:05.669447 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:05.669414 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:05.669447 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:05.669437 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:05.669654 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:05.669561 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:05.669654 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:05.669644 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:05.669785 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:05.669762 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:05.669854 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:05.669838 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:05.878768 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:05.878683 2566 generic.go:356] "Generic (PLEG): container finished" podID="05a67278-34b4-462d-b4cb-9250800bf3db" containerID="1b8c238ee2d4663391c4e99baeba42226991566d44c8896deb8073ed68c10783" exitCode=0 Nov 03 14:11:05.878768 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:05.878739 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99kd6" event={"ID":"05a67278-34b4-462d-b4cb-9250800bf3db","Type":"ContainerDied","Data":"1b8c238ee2d4663391c4e99baeba42226991566d44c8896deb8073ed68c10783"} Nov 03 14:11:06.255161 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:06.255094 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dlwbh_edb1d4a1-0656-4417-8b7d-86a68db196ee/node-ca/0.log" Nov 03 14:11:07.669997 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:07.669820 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:07.670472 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:07.669826 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:07.670472 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:07.670097 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:07.670472 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:07.669850 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:07.670472 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:07.670156 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:07.670472 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:07.670214 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:07.886930 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:07.886896 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99kd6" event={"ID":"05a67278-34b4-462d-b4cb-9250800bf3db","Type":"ContainerStarted","Data":"c448caa6c81bc55ba0a5b992c35a7721ee924452cc0821303b60830e923b5394"} Nov 03 14:11:08.893277 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:08.893246 2566 generic.go:356] "Generic (PLEG): container finished" podID="05a67278-34b4-462d-b4cb-9250800bf3db" containerID="c448caa6c81bc55ba0a5b992c35a7721ee924452cc0821303b60830e923b5394" exitCode=0 Nov 03 14:11:08.893720 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:08.893325 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99kd6" event={"ID":"05a67278-34b4-462d-b4cb-9250800bf3db","Type":"ContainerDied","Data":"c448caa6c81bc55ba0a5b992c35a7721ee924452cc0821303b60830e923b5394"} Nov 03 14:11:08.896591 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:08.896570 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" event={"ID":"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc","Type":"ContainerStarted","Data":"8837cd3dbfe435f419e291c79f0684cc562bb8d3a89fb633204cbaa369a3e007"} Nov 03 14:11:08.896675 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:08.896655 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" event={"ID":"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc","Type":"ContainerStarted","Data":"ad55bc9c942bf2a8e1513aa51b9634e2d78cc3b679c26b6d516647ea7be07af6"} Nov 03 14:11:08.896714 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:08.896691 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" event={"ID":"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc","Type":"ContainerStarted","Data":"9679da2c804b8ddf6bcd4d9a8cef53127b1a61fa0309edc15022c11bbae731d0"} Nov 03 14:11:08.896752 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:08.896727 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" event={"ID":"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc","Type":"ContainerStarted","Data":"4260e5adeca07313ca31633b2ff8ad88da508bf75b43e9655f4689eaea114775"} Nov 03 14:11:08.896794 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:08.896755 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" event={"ID":"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc","Type":"ContainerStarted","Data":"b8e674f0b42fde80803265a78609b2bb1254b7dae7c10774ca366da0e070599d"} Nov 03 14:11:08.896794 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:08.896789 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" event={"ID":"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc","Type":"ContainerStarted","Data":"c56168e44a722e79dab0a7f22be88be7b8c2311649141cd8c43967e6d7e68b95"} Nov 03 14:11:09.390716 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:09.390513 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:09.390878 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:09.390750 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:09.390878 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:09.390652 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:11:09.390878 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:09.390824 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret podName:fc214996-6ee2-4bf3-a2cd-49cd4eceaa09 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:25.390810956 +0000 UTC m=+34.175540186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret") pod "global-pull-secret-syncer-tjsp9" (UID: "fc214996-6ee2-4bf3-a2cd-49cd4eceaa09") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:11:09.390878 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:09.390848 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:11:09.390999 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:09.390894 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs podName:dad8ff83-33fe-4482-927e-8b70b1e72643 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:25.390878662 +0000 UTC m=+34.175607891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs") pod "network-metrics-daemon-r5lc9" (UID: "dad8ff83-33fe-4482-927e-8b70b1e72643") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:11:09.491588 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:09.491556 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dch9b\" (UniqueName: \"kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b\") pod \"network-check-target-phmgc\" (UID: \"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf\") " pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:09.491739 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:09.491697 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:11:09.491739 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:09.491713 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:11:09.491739 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:09.491722 2566 projected.go:196] Error preparing data for projected volume kube-api-access-dch9b for pod openshift-network-diagnostics/network-check-target-phmgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:11:09.491843 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:09.491771 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b podName:6f1c105f-a7fa-4aeb-b8de-a413f5def9bf nodeName:}" failed. No retries permitted until 2026-11-03 14:11:25.491753749 +0000 UTC m=+34.276482978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dch9b" (UniqueName: "kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b") pod "network-check-target-phmgc" (UID: "6f1c105f-a7fa-4aeb-b8de-a413f5def9bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:11:09.669944 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:09.669869 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:09.669944 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:09.669898 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:09.670116 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:09.669968 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:09.670116 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:09.670023 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:09.670116 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:09.670090 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:09.670223 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:09.670192 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:09.901214 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:09.901176 2566 generic.go:356] "Generic (PLEG): container finished" podID="05a67278-34b4-462d-b4cb-9250800bf3db" containerID="fc3d9b67547ebc0e6ae822c6893bcd31c980e9a208f50a154274cdaed19a6302" exitCode=0 Nov 03 14:11:09.901585 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:09.901252 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99kd6" event={"ID":"05a67278-34b4-462d-b4cb-9250800bf3db","Type":"ContainerDied","Data":"fc3d9b67547ebc0e6ae822c6893bcd31c980e9a208f50a154274cdaed19a6302"} Nov 03 14:11:10.907485 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:10.907455 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99kd6" event={"ID":"05a67278-34b4-462d-b4cb-9250800bf3db","Type":"ContainerStarted","Data":"6128ae53e9c8036ca5cbb9b95ed5de908b52305d9bad1181dc2ea5a5e0e5c151"} Nov 03 14:11:10.959293 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:10.959249 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-99kd6" podStartSLOduration=9.929176539 podStartE2EDuration="19.959234571s" totalImagesPullingTime="10.030058032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:10.959050785 +0000 UTC m=+19.743780022" watchObservedRunningTime="2026-11-03 14:11:10.959234571 +0000 UTC m=+19.743963807" Nov 03 14:11:11.670548 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:11.670514 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:11.670729 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:11.670699 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:11.670768 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:11.670718 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:11.670808 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:11.670777 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:11.670808 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:11.670792 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:11.670871 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:11.670852 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:11.912479 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:11.912444 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" event={"ID":"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc","Type":"ContainerStarted","Data":"deb31bfba8de8679b6f7dfff30666b141f0408be6ee3822e88670068921adcb3"} Nov 03 14:11:13.669625 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:13.669571 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:13.669625 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:13.669596 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:13.669625 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:13.669624 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:13.670536 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:13.669705 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:13.670536 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:13.669758 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:13.670536 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:13.669841 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:13.919813 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:13.919741 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" event={"ID":"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc","Type":"ContainerStarted","Data":"18cae2e0deb4fd8e3df3c8cf4c4924c945000f544b74aee54d73dd38b762cb56"} Nov 03 14:11:13.920021 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:13.920001 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:11:13.920093 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:13.920072 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:11:13.936195 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:13.936168 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:11:13.940062 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:13.940043 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:11:14.010449 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:14.010397 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" podStartSLOduration=8.391008852 podStartE2EDuration="23.010383209s" totalImagesPullingTime="14.619374357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:14.009776545 +0000 UTC m=+22.794505781" watchObservedRunningTime="2026-11-03 14:11:14.010383209 +0000 UTC m=+22.795112527" Nov 03 14:11:14.922311 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:14.922276 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 14:11:14.983221 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:14.983191 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:11:15.669489 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:15.669460 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:15.669667 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:15.669518 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:15.669667 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:15.669531 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:15.669667 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:15.669602 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:15.669667 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:15.669641 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:15.669798 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:15.669742 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:16.286765 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:16.286725 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:11:16.287189 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:16.286850 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 14:11:16.287280 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:16.287263 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:11:17.669472 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:17.669220 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:17.669947 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:17.669236 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:17.669947 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:17.669573 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:17.669947 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:17.669236 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:17.669947 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:17.669690 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:17.669947 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:17.669781 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:19.669801 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:19.669768 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:19.669801 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:19.669785 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:19.670350 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:19.669882 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:19.670350 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:19.669945 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:19.670350 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:19.669980 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:19.670350 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:19.670050 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:21.670072 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:21.670046 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:21.670477 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:21.670122 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:21.670477 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:21.670203 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:21.670477 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:21.670294 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:21.670477 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:21.670311 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:21.670477 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:21.670356 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:23.669744 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:23.669704 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:23.669744 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:23.669725 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:23.670226 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:23.669831 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:23.670226 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:23.669935 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:23.670226 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:23.670004 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:23.670226 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:23.670084 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:25.188573 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:25.188544 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-phmgc"] Nov 03 14:11:25.189006 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:25.188721 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:25.189006 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:25.188836 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:25.190945 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:25.190925 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tjsp9"] Nov 03 14:11:25.191066 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:25.190999 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:25.191110 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:25.191074 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:25.194280 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:25.194253 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r5lc9"] Nov 03 14:11:25.194383 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:25.194354 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:25.194484 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:25.194459 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:25.410246 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:25.410210 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:25.410438 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:25.410256 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:25.410438 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:25.410357 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:11:25.410438 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:25.410366 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:11:25.410438 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:25.410414 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret podName:fc214996-6ee2-4bf3-a2cd-49cd4eceaa09 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:57.410397609 +0000 UTC m=+66.195126824 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret") pod "global-pull-secret-syncer-tjsp9" (UID: "fc214996-6ee2-4bf3-a2cd-49cd4eceaa09") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:11:25.410438 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:25.410430 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs podName:dad8ff83-33fe-4482-927e-8b70b1e72643 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:57.410421869 +0000 UTC m=+66.195151083 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs") pod "network-metrics-daemon-r5lc9" (UID: "dad8ff83-33fe-4482-927e-8b70b1e72643") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:11:25.511423 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:25.511335 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dch9b\" (UniqueName: \"kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b\") pod \"network-check-target-phmgc\" (UID: \"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf\") " pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:25.511593 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:25.511482 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:11:25.511593 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:25.511501 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:11:25.511593 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:25.511513 2566 projected.go:196] Error preparing data for projected volume kube-api-access-dch9b for pod openshift-network-diagnostics/network-check-target-phmgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:11:25.511593 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:25.511573 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b podName:6f1c105f-a7fa-4aeb-b8de-a413f5def9bf nodeName:}" failed. No retries permitted until 2026-11-03 14:11:57.511557331 +0000 UTC m=+66.296286560 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dch9b" (UniqueName: "kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b") pod "network-check-target-phmgc" (UID: "6f1c105f-a7fa-4aeb-b8de-a413f5def9bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:11:25.720650 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:25.720620 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qnhkx_b586d272-c9c0-455f-aec9-333ae6d0c69a/dns-node-resolver/0.log" Nov 03 14:11:26.669849 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:26.669818 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:26.669849 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:26.669850 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:26.670268 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:26.669918 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:26.670268 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:26.669957 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:26.670268 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:26.670043 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:26.670268 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:26.670103 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:26.715586 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:26.715558 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dlwbh_edb1d4a1-0656-4417-8b7d-86a68db196ee/node-ca/0.log" Nov 03 14:11:28.669228 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:28.669191 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:28.669653 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:28.669294 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:28.669653 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:28.669307 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:28.669653 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:28.669302 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phmgc" podUID="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" Nov 03 14:11:28.669653 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:28.669383 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjsp9" podUID="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" Nov 03 14:11:28.669653 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:28.669462 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5lc9" podUID="dad8ff83-33fe-4482-927e-8b70b1e72643" Nov 03 14:11:28.978452 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:28.978427 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeReady" Nov 03 14:11:28.978568 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:28.978542 2566 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 14:11:29.053003 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.052971 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j6lgv"] Nov 03 14:11:29.073212 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.073173 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2twr6"] Nov 03 14:11:29.073407 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.073307 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.090417 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.090391 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wqbbs"] Nov 03 14:11:29.090540 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.090527 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2twr6" Nov 03 14:11:29.092625 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.092412 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 14:11:29.094065 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.094000 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghsgd\"" Nov 03 14:11:29.094161 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.094124 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 14:11:29.108664 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.108635 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j6lgv"] Nov 03 14:11:29.108775 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.108674 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2twr6"] Nov 03 14:11:29.108775 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.108689 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqbbs"] Nov 03 14:11:29.108775 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.108766 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.195345 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.195261 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 14:11:29.195345 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.195261 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 14:11:29.195345 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.195327 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 14:11:29.195556 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.195348 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 14:11:29.195556 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.195417 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hlrhs\"" Nov 03 14:11:29.195556 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.195430 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 14:11:29.195556 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.195440 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 14:11:29.195721 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.195569 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 14:11:29.195721 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.195657 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r6xhc\"" Nov 03 14:11:29.237132 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.237111 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2q54\" (UniqueName: \"kubernetes.io/projected/977c941d-36f2-4c16-8396-95bdce627a74-kube-api-access-g2q54\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.237239 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.237169 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c941d-36f2-4c16-8396-95bdce627a74-tmp-dir\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.237289 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.237238 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.237289 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.237269 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c941d-36f2-4c16-8396-95bdce627a74-metrics-tls\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.237375 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.237299 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpp9t\" (UniqueName: \"kubernetes.io/projected/f394a27c-5456-4b99-8b59-b89d25aae03b-kube-api-access-xpp9t\") pod \"ingress-canary-2twr6\" (UID: \"f394a27c-5456-4b99-8b59-b89d25aae03b\") " pod="openshift-ingress-canary/ingress-canary-2twr6" Nov 03 14:11:29.237375 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.237328 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4z4z\" (UniqueName: \"kubernetes.io/projected/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-kube-api-access-h4z4z\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.237458 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.237446 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f394a27c-5456-4b99-8b59-b89d25aae03b-cert\") pod \"ingress-canary-2twr6\" (UID: \"f394a27c-5456-4b99-8b59-b89d25aae03b\") " pod="openshift-ingress-canary/ingress-canary-2twr6" Nov 03 14:11:29.237503 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.237476 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-crio-socket\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.237503 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.237492 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-data-volume\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.237577 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.237533 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/977c941d-36f2-4c16-8396-95bdce627a74-config-volume\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.237630 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.237592 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.338851 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.338822 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2q54\" (UniqueName: \"kubernetes.io/projected/977c941d-36f2-4c16-8396-95bdce627a74-kube-api-access-g2q54\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.338954 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.338871 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c941d-36f2-4c16-8396-95bdce627a74-tmp-dir\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.338954 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.338891 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.338954 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.338912 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c941d-36f2-4c16-8396-95bdce627a74-metrics-tls\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.338954 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.338946 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpp9t\" (UniqueName: \"kubernetes.io/projected/f394a27c-5456-4b99-8b59-b89d25aae03b-kube-api-access-xpp9t\") pod \"ingress-canary-2twr6\" (UID: \"f394a27c-5456-4b99-8b59-b89d25aae03b\") " pod="openshift-ingress-canary/ingress-canary-2twr6" Nov 03 14:11:29.339150 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.338985 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4z4z\" (UniqueName: \"kubernetes.io/projected/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-kube-api-access-h4z4z\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.339243 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.339218 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f394a27c-5456-4b99-8b59-b89d25aae03b-cert\") pod \"ingress-canary-2twr6\" (UID: \"f394a27c-5456-4b99-8b59-b89d25aae03b\") " pod="openshift-ingress-canary/ingress-canary-2twr6" Nov 03 14:11:29.339295 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.339273 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-crio-socket\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.339295 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.339226 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c941d-36f2-4c16-8396-95bdce627a74-tmp-dir\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.339384 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.339301 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-data-volume\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.339384 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.339324 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/977c941d-36f2-4c16-8396-95bdce627a74-config-volume\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.339384 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.339363 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.339529 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.339459 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-crio-socket\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.339722 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.339696 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-data-volume\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.339956 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.339939 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.340034 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.339954 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/977c941d-36f2-4c16-8396-95bdce627a74-config-volume\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.343177 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.343023 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c941d-36f2-4c16-8396-95bdce627a74-metrics-tls\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.343277 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.343108 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.343330 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.343281 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f394a27c-5456-4b99-8b59-b89d25aae03b-cert\") pod \"ingress-canary-2twr6\" (UID: \"f394a27c-5456-4b99-8b59-b89d25aae03b\") " pod="openshift-ingress-canary/ingress-canary-2twr6" Nov 03 14:11:29.362159 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.362133 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4z4z\" (UniqueName: \"kubernetes.io/projected/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-kube-api-access-h4z4z\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.362564 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.362542 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2q54\" (UniqueName: \"kubernetes.io/projected/977c941d-36f2-4c16-8396-95bdce627a74-kube-api-access-g2q54\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.364072 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.364056 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpp9t\" (UniqueName: \"kubernetes.io/projected/f394a27c-5456-4b99-8b59-b89d25aae03b-kube-api-access-xpp9t\") pod \"ingress-canary-2twr6\" (UID: \"f394a27c-5456-4b99-8b59-b89d25aae03b\") " pod="openshift-ingress-canary/ingress-canary-2twr6" Nov 03 14:11:29.408885 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.408848 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:29.423000 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.422972 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2twr6" Nov 03 14:11:29.462671 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.462647 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:11:29.804687 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.804668 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2twr6"] Nov 03 14:11:29.807914 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.807890 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j6lgv"] Nov 03 14:11:29.808780 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.808625 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqbbs"] Nov 03 14:11:29.810566 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:11:29.810543 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977c941d_36f2_4c16_8396_95bdce627a74.slice/crio-03341514a396ff5f2bd42621fbf5e52b36887ce841edb6497040a8fa71e68566 WatchSource:0}: Error finding container 03341514a396ff5f2bd42621fbf5e52b36887ce841edb6497040a8fa71e68566: Status 404 returned error can't find the container with id 03341514a396ff5f2bd42621fbf5e52b36887ce841edb6497040a8fa71e68566 Nov 03 14:11:29.811268 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:11:29.811246 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf824baea_c6ee_409c_a1d3_e4f8b02bc6c5.slice/crio-0483756c8af80f00eb23edb5bc09c749cfc1dd101501f56ccd30bf6498439cc6 WatchSource:0}: Error finding container 0483756c8af80f00eb23edb5bc09c749cfc1dd101501f56ccd30bf6498439cc6: Status 404 returned error can't find the container with id 0483756c8af80f00eb23edb5bc09c749cfc1dd101501f56ccd30bf6498439cc6 Nov 03 14:11:29.955058 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.955024 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqbbs" event={"ID":"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5","Type":"ContainerStarted","Data":"cb715ea9642a8ac66d0431ec310857ea081595dcd1d9544d5e434eff6bc9c8a5"} Nov 03 14:11:29.955210 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.955087 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqbbs" event={"ID":"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5","Type":"ContainerStarted","Data":"0483756c8af80f00eb23edb5bc09c749cfc1dd101501f56ccd30bf6498439cc6"} Nov 03 14:11:29.956124 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.956102 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2twr6" event={"ID":"f394a27c-5456-4b99-8b59-b89d25aae03b","Type":"ContainerStarted","Data":"2cd11b588eab8bb8eb0a93c647495527199429a62ff928e732585039a780f536"} Nov 03 14:11:29.957245 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:29.957226 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j6lgv" event={"ID":"977c941d-36f2-4c16-8396-95bdce627a74","Type":"ContainerStarted","Data":"03341514a396ff5f2bd42621fbf5e52b36887ce841edb6497040a8fa71e68566"} Nov 03 14:11:30.669913 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:30.669875 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:30.670346 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:30.670330 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:30.670781 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:30.670768 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:30.695006 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:30.694927 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 14:11:30.697264 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:30.696049 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 14:11:30.708276 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:30.708244 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ml487\"" Nov 03 14:11:30.708380 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:30.708323 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rmj5t\"" Nov 03 14:11:30.708380 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:30.708301 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 14:11:30.708476 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:30.708388 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bgwlk\"" Nov 03 14:11:30.721226 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:30.721204 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 14:11:31.014014 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.013097 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph"] Nov 03 14:11:31.029333 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.029307 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.048968 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.048946 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph"] Nov 03 14:11:31.055426 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.055310 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 14:11:31.055426 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.055370 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 14:11:31.055599 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.055561 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 14:11:31.058665 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.058643 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 14:11:31.060407 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.060389 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bvxcr\"" Nov 03 14:11:31.060996 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.060918 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 14:11:31.101444 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.101408 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7"] Nov 03 14:11:31.118650 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.117868 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.125822 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.125681 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zbs97"] Nov 03 14:11:31.138595 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.138570 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7"] Nov 03 14:11:31.138798 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.138783 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.145492 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.145433 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rp88m\"" Nov 03 14:11:31.145492 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.145433 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 14:11:31.145685 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.145526 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 14:11:31.153282 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.153261 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 14:11:31.154835 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.154788 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e380172-2ed6-4bd3-9fdb-83a694b400d0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.154945 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.154856 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhgqh\" (UniqueName: \"kubernetes.io/projected/8e380172-2ed6-4bd3-9fdb-83a694b400d0-kube-api-access-jhgqh\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.155067 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.155034 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e380172-2ed6-4bd3-9fdb-83a694b400d0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.155140 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.155102 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 14:11:31.155140 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.155107 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e380172-2ed6-4bd3-9fdb-83a694b400d0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.156653 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.156602 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 14:11:31.156778 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.156758 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 14:11:31.156870 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.156832 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zjlcb\"" Nov 03 14:11:31.256313 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.256137 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.256313 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.256186 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.256535 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.256311 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.256535 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.256358 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.256535 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.256453 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-sys\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.256535 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.256491 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-metrics-client-ca\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.256715 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.256556 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e380172-2ed6-4bd3-9fdb-83a694b400d0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.256715 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.256655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e380172-2ed6-4bd3-9fdb-83a694b400d0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.256715 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.256685 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-root\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.256854 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.256758 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-wtmp\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.256854 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.256799 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47v7j\" (UniqueName: \"kubernetes.io/projected/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-api-access-47v7j\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.257128 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.256898 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.257248 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.257153 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.257248 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.257189 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-textfile\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.257248 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.257220 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-accelerators-collector-config\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.257418 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.257252 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e380172-2ed6-4bd3-9fdb-83a694b400d0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.257418 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.257284 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2wbc\" (UniqueName: \"kubernetes.io/projected/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-kube-api-access-f2wbc\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.257418 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.257312 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhgqh\" (UniqueName: \"kubernetes.io/projected/8e380172-2ed6-4bd3-9fdb-83a694b400d0-kube-api-access-jhgqh\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.257418 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.257337 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-tls\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.257418 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.257402 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e380172-2ed6-4bd3-9fdb-83a694b400d0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.260879 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.260852 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e380172-2ed6-4bd3-9fdb-83a694b400d0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.261014 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.260997 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e380172-2ed6-4bd3-9fdb-83a694b400d0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.282445 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.282416 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhgqh\" (UniqueName: \"kubernetes.io/projected/8e380172-2ed6-4bd3-9fdb-83a694b400d0-kube-api-access-jhgqh\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.357790 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.357752 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-wtmp\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.357790 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.357800 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47v7j\" (UniqueName: \"kubernetes.io/projected/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-api-access-47v7j\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.358012 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.357831 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.358012 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.357869 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.358012 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.357905 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-textfile\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.358012 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.357936 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-accelerators-collector-config\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.358012 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.357955 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-wtmp\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.358012 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.357967 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wbc\" (UniqueName: \"kubernetes.io/projected/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-kube-api-access-f2wbc\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.358012 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.357994 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-tls\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.358212 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.358017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.358212 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.358042 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.358212 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.358079 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.358212 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.358104 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.358212 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.358130 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-sys\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.358212 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.358148 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-metrics-client-ca\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.358212 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.358168 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-root\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.359008 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.358902 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-textfile\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.359008 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.358909 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-root\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.359975 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.359486 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-sys\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.359975 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.359692 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-metrics-client-ca\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.359975 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.359858 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.371163 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.371113 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-accelerators-collector-config\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.371856 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.371837 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:11:31.372948 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.372869 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-tls\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.372948 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.372925 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.373578 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.373397 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.373699 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.373552 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.374173 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.374153 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.374791 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.374770 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.381669 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.381647 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wbc\" (UniqueName: \"kubernetes.io/projected/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-kube-api-access-f2wbc\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.381795 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.381774 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47v7j\" (UniqueName: \"kubernetes.io/projected/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-api-access-47v7j\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.473415 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.473190 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:11:31.525013 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.524919 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:11:31.746344 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.746266 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph"] Nov 03 14:11:31.790248 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:11:31.790181 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e380172_2ed6_4bd3_9fdb_83a694b400d0.slice/crio-0a431ebd473676f2458785a54cd60c6116d7c401339a8f5af2cb6dbbdac4973e WatchSource:0}: Error finding container 0a431ebd473676f2458785a54cd60c6116d7c401339a8f5af2cb6dbbdac4973e: Status 404 returned error can't find the container with id 0a431ebd473676f2458785a54cd60c6116d7c401339a8f5af2cb6dbbdac4973e Nov 03 14:11:31.800805 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.800784 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7"] Nov 03 14:11:31.965798 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.965766 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqbbs" event={"ID":"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5","Type":"ContainerStarted","Data":"fbb30f012ff2cfd1fc85a5bfb98013ed5c239aca0622a4edf78391ff86229e0b"} Nov 03 14:11:31.968171 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.967470 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zbs97" event={"ID":"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea","Type":"ContainerStarted","Data":"65ffbf449cc14e3ad047a0449e575b697995b6fcfc4f8f8c73374c57c94918fa"} Nov 03 14:11:31.968838 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.968814 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" event={"ID":"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56","Type":"ContainerStarted","Data":"f0e2dd814d101ad57d106df1be9fca646bfeafb1c40f40b76f8f91a0eb953367"} Nov 03 14:11:31.970331 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.970306 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" event={"ID":"8e380172-2ed6-4bd3-9fdb-83a694b400d0","Type":"ContainerStarted","Data":"0ad0dff2e9293444046448ac0a87083cc1cac09fb38c3b738414bdc531318e59"} Nov 03 14:11:31.970403 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:31.970351 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" event={"ID":"8e380172-2ed6-4bd3-9fdb-83a694b400d0","Type":"ContainerStarted","Data":"0a431ebd473676f2458785a54cd60c6116d7c401339a8f5af2cb6dbbdac4973e"} Nov 03 14:11:32.975481 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:32.975264 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" event={"ID":"8e380172-2ed6-4bd3-9fdb-83a694b400d0","Type":"ContainerStarted","Data":"bebed036e164d23c4972cec879e8a9ed49dba7e942b474eacefc51206a8491a4"} Nov 03 14:11:32.977449 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:32.977422 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j6lgv" event={"ID":"977c941d-36f2-4c16-8396-95bdce627a74","Type":"ContainerStarted","Data":"c4fcf32f7435c3de10973782808b02ece541075601a3ca41e35f2d52e17db63b"} Nov 03 14:11:32.977566 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:32.977464 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j6lgv" event={"ID":"977c941d-36f2-4c16-8396-95bdce627a74","Type":"ContainerStarted","Data":"2d4804773868ced738b44be43794e5d75d6737dc5f0f34ebca1780a442039ba5"} Nov 03 14:11:32.977662 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:32.977561 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:33.022540 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:33.022485 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-j6lgv" podStartSLOduration=1.657053952 podStartE2EDuration="4.022466521s" totalImagesPullingTime="2.365412569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:33.021761098 +0000 UTC m=+41.806490336" watchObservedRunningTime="2026-11-03 14:11:33.022466521 +0000 UTC m=+41.807195759" Nov 03 14:11:34.984564 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:34.984466 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2twr6" event={"ID":"f394a27c-5456-4b99-8b59-b89d25aae03b","Type":"ContainerStarted","Data":"b3ebd57eae68fc8a0079b71ab5e12b22d3133d9e7802e11943bc3902331e1214"} Nov 03 14:11:34.986247 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:34.986220 2566 generic.go:356] "Generic (PLEG): container finished" podID="b57fa4d4-1e9d-4a56-aba9-c6314ff12aea" containerID="3e0b84076fdb124176e5b50f121f772b65c38be0fbf4f778a4a8dcf2bc6c1c3b" exitCode=0 Nov 03 14:11:34.986354 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:34.986306 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zbs97" event={"ID":"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea","Type":"ContainerDied","Data":"3e0b84076fdb124176e5b50f121f772b65c38be0fbf4f778a4a8dcf2bc6c1c3b"} Nov 03 14:11:34.988768 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:34.988742 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" event={"ID":"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56","Type":"ContainerStarted","Data":"5397ef351272614ae45169ee42026746315ea4ce2f774965f5120628183f6975"} Nov 03 14:11:34.988872 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:34.988810 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" event={"ID":"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56","Type":"ContainerStarted","Data":"520eb6ae93f00926df721247a92e2e6381cd1dabdc6f1b367dbbb0f5d1794650"} Nov 03 14:11:34.988872 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:34.988840 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" event={"ID":"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56","Type":"ContainerStarted","Data":"e96618ac111a1d9d8d0d9f2258ca7bd6ff0e336f95dd9249329e547be411ac25"} Nov 03 14:11:34.991402 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:34.991382 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" event={"ID":"8e380172-2ed6-4bd3-9fdb-83a694b400d0","Type":"ContainerStarted","Data":"8243aaeee1c094082272fb263c2fea336ca63c6f12a3bd6fcbe2cffd10d77996"} Nov 03 14:11:34.998779 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:34.998755 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqbbs" event={"ID":"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5","Type":"ContainerStarted","Data":"d67aec7ac2bdf168a29e06942320d249082a7f7a474a9995b0c5a5487ec3f3e8"} Nov 03 14:11:35.031515 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:35.031473 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2twr6" podStartSLOduration=1.700002187 podStartE2EDuration="6.031459743s" totalImagesPullingTime="4.331457556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:35.030932021 +0000 UTC m=+43.815661258" watchObservedRunningTime="2026-11-03 14:11:35.031459743 +0000 UTC m=+43.816188977" Nov 03 14:11:35.079310 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:35.079262 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wqbbs" podStartSLOduration=1.9719645689999998 podStartE2EDuration="6.079248102s" totalImagesPullingTime="4.107283533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:11:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:35.07851765 +0000 UTC m=+43.863246902" watchObservedRunningTime="2026-11-03 14:11:35.079248102 +0000 UTC m=+43.863977338" Nov 03 14:11:35.126111 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:35.126046 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" podStartSLOduration=1.510408784 podStartE2EDuration="4.12602695s" totalImagesPullingTime="2.615618166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:35.125454049 +0000 UTC m=+43.910183285" watchObservedRunningTime="2026-11-03 14:11:35.12602695 +0000 UTC m=+43.910756189" Nov 03 14:11:35.175649 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:35.175579 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" podStartSLOduration=3.030023084 podStartE2EDuration="5.175564174s" totalImagesPullingTime="2.14554109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:35.174650873 +0000 UTC m=+43.959380106" watchObservedRunningTime="2026-11-03 14:11:35.175564174 +0000 UTC m=+43.960293414" Nov 03 14:11:36.004320 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.004277 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zbs97" event={"ID":"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea","Type":"ContainerStarted","Data":"0631d20f4a3742754aacd984f7a4da73735fc33b50a8f9dcfb07df4f0da20e60"} Nov 03 14:11:36.004745 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.004340 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zbs97" event={"ID":"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea","Type":"ContainerStarted","Data":"6843e5ef3b84970f5a832ad831ce7891d113a9ce1fd6f0aef9155f65c46785d1"} Nov 03 14:11:36.051785 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.051735 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zbs97" podStartSLOduration=2.360747318 podStartE2EDuration="5.051720021s" totalImagesPullingTime="2.579189336s" totalInitContainerRuntime="111.783367ms" isStatefulPod=true podCreationTimestamp="2026-11-03 14:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:36.051131867 +0000 UTC m=+44.835861103" watchObservedRunningTime="2026-11-03 14:11:36.051720021 +0000 UTC m=+44.836449258" Nov 03 14:11:36.341541 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.341464 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-65979fb8bc-sh6c8"] Nov 03 14:11:36.344023 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.344004 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.363492 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.363465 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 14:11:36.363492 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.363483 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 14:11:36.363656 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.363465 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 14:11:36.363695 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.363655 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kfj2l\"" Nov 03 14:11:36.363762 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.363748 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 14:11:36.375665 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.375645 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 14:11:36.389523 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.389505 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 14:11:36.390680 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.390660 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65979fb8bc-sh6c8"] Nov 03 14:11:36.507228 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.507197 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.507405 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.507261 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-metrics-client-ca\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.507405 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.507293 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-telemeter-client-tls\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.507405 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.507364 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-serving-certs-ca-bundle\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.507498 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.507442 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxcm\" (UniqueName: \"kubernetes.io/projected/608de1a1-c806-48bb-aecc-def971eca078-kube-api-access-tjxcm\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.507498 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.507493 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.507555 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.507512 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-federate-client-tls\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.507555 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.507539 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-secret-telemeter-client\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.608065 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.607979 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-secret-telemeter-client\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.608065 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.608031 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.608287 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.608084 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-metrics-client-ca\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.608287 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.608168 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-telemeter-client-tls\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.608287 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.608235 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-serving-certs-ca-bundle\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.608287 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.608281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxcm\" (UniqueName: \"kubernetes.io/projected/608de1a1-c806-48bb-aecc-def971eca078-kube-api-access-tjxcm\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.608631 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.608315 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.608631 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.608344 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-federate-client-tls\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.609175 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.609150 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-metrics-client-ca\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.609274 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.609210 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.609337 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.609316 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-serving-certs-ca-bundle\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.612565 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.612540 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-federate-client-tls\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.612695 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.612626 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.612913 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.612698 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-secret-telemeter-client\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.612913 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.612740 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-telemeter-client-tls\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.631519 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.631493 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxcm\" (UniqueName: \"kubernetes.io/projected/608de1a1-c806-48bb-aecc-def971eca078-kube-api-access-tjxcm\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.657410 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.657368 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:11:36.986523 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:36.986494 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65979fb8bc-sh6c8"] Nov 03 14:11:36.987881 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:11:36.987849 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608de1a1_c806_48bb_aecc_def971eca078.slice/crio-71026b6e406cc80284587c4bfa21304cca3450f73ff547b89b9b085d4d4a2898 WatchSource:0}: Error finding container 71026b6e406cc80284587c4bfa21304cca3450f73ff547b89b9b085d4d4a2898: Status 404 returned error can't find the container with id 71026b6e406cc80284587c4bfa21304cca3450f73ff547b89b9b085d4d4a2898 Nov 03 14:11:37.008779 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:37.008753 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" event={"ID":"608de1a1-c806-48bb-aecc-def971eca078","Type":"ContainerStarted","Data":"71026b6e406cc80284587c4bfa21304cca3450f73ff547b89b9b085d4d4a2898"} Nov 03 14:11:39.016815 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:39.016782 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" event={"ID":"608de1a1-c806-48bb-aecc-def971eca078","Type":"ContainerStarted","Data":"154057be49d4598b0792baf94c9c095273923579bfb892f9748152e8c4702af7"} Nov 03 14:11:41.024880 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:41.024845 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" event={"ID":"608de1a1-c806-48bb-aecc-def971eca078","Type":"ContainerStarted","Data":"9c5f85996b95a32872ec7b0873bf7cc8ada4502ed3e1e1c7e67cb1b9a1299ca3"} Nov 03 14:11:41.025345 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:41.024906 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" event={"ID":"608de1a1-c806-48bb-aecc-def971eca078","Type":"ContainerStarted","Data":"f83b1237bb2f6d06c2f5473b74752dcb8752e645219b161d8651311802ec4038"} Nov 03 14:11:41.103689 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:41.103631 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" podStartSLOduration=2.161707307 podStartE2EDuration="5.103602616s" totalImagesPullingTime="2.941895309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:41.089750479 +0000 UTC m=+49.874479707" watchObservedRunningTime="2026-11-03 14:11:41.103602616 +0000 UTC m=+49.888331853" Nov 03 14:11:42.982350 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:42.982323 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j6lgv" Nov 03 14:11:43.366099 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.366020 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-dzpfq"] Nov 03 14:11:43.369306 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.369285 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:11:43.392930 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.392900 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-dzpfq"] Nov 03 14:11:43.404282 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.404257 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-cv9df\"" Nov 03 14:11:43.404431 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.404303 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 14:11:43.404431 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.404323 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 14:11:43.466199 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.466169 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0694b8ee-14f6-495d-a662-8f06bdfe1c66-tmp\") pod \"downloads-6dd789c8b7-dzpfq\" (UID: \"0694b8ee-14f6-495d-a662-8f06bdfe1c66\") " pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:11:43.466362 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.466211 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7d5k\" (UniqueName: \"kubernetes.io/projected/0694b8ee-14f6-495d-a662-8f06bdfe1c66-kube-api-access-m7d5k\") pod \"downloads-6dd789c8b7-dzpfq\" (UID: \"0694b8ee-14f6-495d-a662-8f06bdfe1c66\") " pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:11:43.567534 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.567498 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7d5k\" (UniqueName: \"kubernetes.io/projected/0694b8ee-14f6-495d-a662-8f06bdfe1c66-kube-api-access-m7d5k\") pod \"downloads-6dd789c8b7-dzpfq\" (UID: \"0694b8ee-14f6-495d-a662-8f06bdfe1c66\") " pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:11:43.567698 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.567666 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0694b8ee-14f6-495d-a662-8f06bdfe1c66-tmp\") pod \"downloads-6dd789c8b7-dzpfq\" (UID: \"0694b8ee-14f6-495d-a662-8f06bdfe1c66\") " pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:11:43.568098 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.568075 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0694b8ee-14f6-495d-a662-8f06bdfe1c66-tmp\") pod \"downloads-6dd789c8b7-dzpfq\" (UID: \"0694b8ee-14f6-495d-a662-8f06bdfe1c66\") " pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:11:43.589672 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.589646 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7d5k\" (UniqueName: \"kubernetes.io/projected/0694b8ee-14f6-495d-a662-8f06bdfe1c66-kube-api-access-m7d5k\") pod \"downloads-6dd789c8b7-dzpfq\" (UID: \"0694b8ee-14f6-495d-a662-8f06bdfe1c66\") " pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:11:43.681146 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:43.681117 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:11:44.014883 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:44.014858 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-dzpfq"] Nov 03 14:11:44.017587 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:11:44.017555 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0694b8ee_14f6_495d_a662_8f06bdfe1c66.slice/crio-040234a2b252b62d917e66263b509fcf4e1f7886f408e65d77e776a2dbff2fc9 WatchSource:0}: Error finding container 040234a2b252b62d917e66263b509fcf4e1f7886f408e65d77e776a2dbff2fc9: Status 404 returned error can't find the container with id 040234a2b252b62d917e66263b509fcf4e1f7886f408e65d77e776a2dbff2fc9 Nov 03 14:11:44.037079 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:44.037047 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-dzpfq" event={"ID":"0694b8ee-14f6-495d-a662-8f06bdfe1c66","Type":"ContainerStarted","Data":"040234a2b252b62d917e66263b509fcf4e1f7886f408e65d77e776a2dbff2fc9"} Nov 03 14:11:46.938311 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:46.938282 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:11:48.045457 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.045426 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-84c65c5d7-j9sv7"] Nov 03 14:11:48.050157 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.050131 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.079441 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.079409 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fn7sx\"" Nov 03 14:11:48.079633 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.079452 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 14:11:48.079633 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.079489 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 14:11:48.079633 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.079455 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 14:11:48.079633 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.079519 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 14:11:48.079633 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.079523 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 14:11:48.081741 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.081720 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c65c5d7-j9sv7"] Nov 03 14:11:48.117929 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.117898 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-oauth-config\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.118118 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.117934 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-service-ca\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.118118 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.117962 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61d8c282-03cc-4a20-a999-6cda0b14c87e-tmp\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.118118 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.118089 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-oauth-serving-cert\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.118260 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.118127 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-serving-cert\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.118260 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.118249 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j5qj\" (UniqueName: \"kubernetes.io/projected/61d8c282-03cc-4a20-a999-6cda0b14c87e-kube-api-access-7j5qj\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.118326 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.118312 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-config\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.219360 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.219316 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-serving-cert\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.219536 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.219383 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j5qj\" (UniqueName: \"kubernetes.io/projected/61d8c282-03cc-4a20-a999-6cda0b14c87e-kube-api-access-7j5qj\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.219536 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.219448 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-config\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.219536 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.219476 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-oauth-config\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.219536 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.219502 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-service-ca\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.219758 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.219540 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61d8c282-03cc-4a20-a999-6cda0b14c87e-tmp\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.219758 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.219590 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-oauth-serving-cert\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.220146 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.220063 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61d8c282-03cc-4a20-a999-6cda0b14c87e-tmp\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.220344 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.220202 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-oauth-serving-cert\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.220344 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.220201 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-config\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.220848 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.220767 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-service-ca\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.222250 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.222223 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-serving-cert\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.222627 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.222590 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-oauth-config\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.245519 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.245490 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j5qj\" (UniqueName: \"kubernetes.io/projected/61d8c282-03cc-4a20-a999-6cda0b14c87e-kube-api-access-7j5qj\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.364629 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.364531 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:48.681215 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:48.681183 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c65c5d7-j9sv7"] Nov 03 14:11:48.683010 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:11:48.682979 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d8c282_03cc_4a20_a999_6cda0b14c87e.slice/crio-d9a77ad916f5ee90d19fd297849f1a6e4f9be2ee781e2be5d061d74572af8362 WatchSource:0}: Error finding container d9a77ad916f5ee90d19fd297849f1a6e4f9be2ee781e2be5d061d74572af8362: Status 404 returned error can't find the container with id d9a77ad916f5ee90d19fd297849f1a6e4f9be2ee781e2be5d061d74572af8362 Nov 03 14:11:49.060511 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:49.060432 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c65c5d7-j9sv7" event={"ID":"61d8c282-03cc-4a20-a999-6cda0b14c87e","Type":"ContainerStarted","Data":"d9a77ad916f5ee90d19fd297849f1a6e4f9be2ee781e2be5d061d74572af8362"} Nov 03 14:11:52.073523 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:52.073488 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c65c5d7-j9sv7" event={"ID":"61d8c282-03cc-4a20-a999-6cda0b14c87e","Type":"ContainerStarted","Data":"4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920"} Nov 03 14:11:52.121700 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:52.121572 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-84c65c5d7-j9sv7" podStartSLOduration=0.97573986 podStartE2EDuration="4.121557044s" totalImagesPullingTime="3.145817184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:52.12050237 +0000 UTC m=+60.905231612" watchObservedRunningTime="2026-11-03 14:11:52.121557044 +0000 UTC m=+60.906286275" Nov 03 14:11:57.506256 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.506214 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:57.506852 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.506276 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:57.521240 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.521208 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 14:11:57.529032 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.529009 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:57.534271 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.534249 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 14:11:57.539575 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.539551 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:57.607404 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.607371 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dch9b\" (UniqueName: \"kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b\") pod \"network-check-target-phmgc\" (UID: \"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf\") " pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:57.622252 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.622223 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 14:11:57.642932 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.642897 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 14:11:57.651759 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.651727 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dch9b\" (UniqueName: \"kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b\") pod \"network-check-target-phmgc\" (UID: \"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf\") " pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:57.714155 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.714120 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bgwlk\"" Nov 03 14:11:57.720210 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.720188 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:11:57.726282 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.726263 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ml487\"" Nov 03 14:11:57.732318 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.732298 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:11:57.741219 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.741194 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rmj5t\"" Nov 03 14:11:57.746758 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:57.746736 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:11:58.033509 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:58.033482 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-phmgc"] Nov 03 14:11:58.035385 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:11:58.035359 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1c105f_a7fa_4aeb_b8de_a413f5def9bf.slice/crio-03a2ab831fb7262165c5cdc198b059eb0636bc1c1e20be57ffb5bda465174c8a WatchSource:0}: Error finding container 03a2ab831fb7262165c5cdc198b059eb0636bc1c1e20be57ffb5bda465174c8a: Status 404 returned error can't find the container with id 03a2ab831fb7262165c5cdc198b059eb0636bc1c1e20be57ffb5bda465174c8a Nov 03 14:11:58.069729 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:58.069703 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tjsp9"] Nov 03 14:11:58.071787 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:11:58.071754 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc214996_6ee2_4bf3_a2cd_49cd4eceaa09.slice/crio-bb50e172753912610fb1cffdee52e795180f08135ce69cc8d4f92cdbd470dc70 WatchSource:0}: Error finding container bb50e172753912610fb1cffdee52e795180f08135ce69cc8d4f92cdbd470dc70: Status 404 returned error can't find the container with id bb50e172753912610fb1cffdee52e795180f08135ce69cc8d4f92cdbd470dc70 Nov 03 14:11:58.072785 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:58.072754 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r5lc9"] Nov 03 14:11:58.074326 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:11:58.074301 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad8ff83_33fe_4482_927e_8b70b1e72643.slice/crio-f6282608a6482faf891a0cf254501636a13b0138457ce64ca15f934ebc1710f0 WatchSource:0}: Error finding container f6282608a6482faf891a0cf254501636a13b0138457ce64ca15f934ebc1710f0: Status 404 returned error can't find the container with id f6282608a6482faf891a0cf254501636a13b0138457ce64ca15f934ebc1710f0 Nov 03 14:11:58.094484 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:58.094453 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tjsp9" event={"ID":"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09","Type":"ContainerStarted","Data":"bb50e172753912610fb1cffdee52e795180f08135ce69cc8d4f92cdbd470dc70"} Nov 03 14:11:58.095835 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:58.095806 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-phmgc" event={"ID":"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf","Type":"ContainerStarted","Data":"03a2ab831fb7262165c5cdc198b059eb0636bc1c1e20be57ffb5bda465174c8a"} Nov 03 14:11:58.100218 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:58.100134 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5lc9" event={"ID":"dad8ff83-33fe-4482-927e-8b70b1e72643","Type":"ContainerStarted","Data":"f6282608a6482faf891a0cf254501636a13b0138457ce64ca15f934ebc1710f0"} Nov 03 14:11:58.365054 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:58.364961 2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:58.365220 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:58.365119 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:58.370480 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:58.370456 2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:59.111067 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:11:59.110867 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:11:59.498528 ip-10-0-130-191 kubenswrapper[2566]: E1103 14:11:59.498485 2566 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:12:00.049588 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.049558 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c959ffc98-lphmd"] Nov 03 14:12:00.055696 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.055661 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.076120 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.076093 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c959ffc98-lphmd"] Nov 03 14:12:00.114140 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.114028 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5lc9" event={"ID":"dad8ff83-33fe-4482-927e-8b70b1e72643","Type":"ContainerStarted","Data":"f49c0022656062bdc8f86bcd803f54a4fe712f65c0ab945d77f9e56ef479a70a"} Nov 03 14:12:00.114140 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.114094 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5lc9" event={"ID":"dad8ff83-33fe-4482-927e-8b70b1e72643","Type":"ContainerStarted","Data":"64233d591bd49218fa0b96d2716c70ab10f1a8aec6e6a92631589f0602e690a0"} Nov 03 14:12:00.114863 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.114745 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 14:12:00.127970 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.127941 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a22219af-1262-42a9-8cbe-8e6323e4375b-tmp\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.128152 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.127992 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-console-config\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.128152 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.128090 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd8sm\" (UniqueName: \"kubernetes.io/projected/a22219af-1262-42a9-8cbe-8e6323e4375b-kube-api-access-wd8sm\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.128278 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.128155 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-oauth-config\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.128278 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.128182 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-service-ca\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.128278 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.128220 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-trusted-ca-bundle\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.128278 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.128263 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-oauth-serving-cert\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.128467 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.128292 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-serving-cert\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.166458 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.165677 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r5lc9" podStartSLOduration=68.087671832 podStartE2EDuration="1m9.165656774s" totalImagesPullingTime="1.077984942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:00.164328027 +0000 UTC m=+68.949057266" watchObservedRunningTime="2026-11-03 14:12:00.165656774 +0000 UTC m=+68.950386020" Nov 03 14:12:00.228821 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.228777 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-oauth-config\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.228821 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.228820 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-service-ca\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.229072 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.228857 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-trusted-ca-bundle\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.229072 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.228945 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-oauth-serving-cert\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.229072 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.228976 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-serving-cert\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.229072 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.229030 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a22219af-1262-42a9-8cbe-8e6323e4375b-tmp\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.229072 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.229069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-console-config\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.229314 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.229116 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8sm\" (UniqueName: \"kubernetes.io/projected/a22219af-1262-42a9-8cbe-8e6323e4375b-kube-api-access-wd8sm\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.231124 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.231097 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-oauth-serving-cert\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.232458 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.232432 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-trusted-ca-bundle\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.232780 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.232575 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a22219af-1262-42a9-8cbe-8e6323e4375b-tmp\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.233095 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.233019 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-console-config\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.233586 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.233253 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-service-ca\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.234347 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.234325 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-oauth-config\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.236403 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.236365 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-serving-cert\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.255075 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.255046 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8sm\" (UniqueName: \"kubernetes.io/projected/a22219af-1262-42a9-8cbe-8e6323e4375b-kube-api-access-wd8sm\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.372031 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.371954 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:00.696743 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:00.696673 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c959ffc98-lphmd"] Nov 03 14:12:00.744801 ip-10-0-130-191 kubenswrapper[2566]: W1103 14:12:00.744764 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22219af_1262_42a9_8cbe_8e6323e4375b.slice/crio-fafcf58717120bce6dc686e7b602493ee4ad8879d233785ca2b2bf90c909e84e WatchSource:0}: Error finding container fafcf58717120bce6dc686e7b602493ee4ad8879d233785ca2b2bf90c909e84e: Status 404 returned error can't find the container with id fafcf58717120bce6dc686e7b602493ee4ad8879d233785ca2b2bf90c909e84e Nov 03 14:12:01.119190 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:01.119004 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c959ffc98-lphmd" event={"ID":"a22219af-1262-42a9-8cbe-8e6323e4375b","Type":"ContainerStarted","Data":"0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3"} Nov 03 14:12:01.119190 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:01.119070 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c959ffc98-lphmd" event={"ID":"a22219af-1262-42a9-8cbe-8e6323e4375b","Type":"ContainerStarted","Data":"fafcf58717120bce6dc686e7b602493ee4ad8879d233785ca2b2bf90c909e84e"} Nov 03 14:12:01.168864 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:01.168805 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c959ffc98-lphmd" podStartSLOduration=1.168785188 podStartE2EDuration="1.168785188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:01.167059013 +0000 UTC m=+69.951788245" watchObservedRunningTime="2026-11-03 14:12:01.168785188 +0000 UTC m=+69.953514418" Nov 03 14:12:02.126467 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:02.126433 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-dzpfq" event={"ID":"0694b8ee-14f6-495d-a662-8f06bdfe1c66","Type":"ContainerStarted","Data":"e91f3884f812f6c6fffa9bef8d9540efb90534f3279f378517462ae0dc24ec79"} Nov 03 14:12:02.126947 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:02.126759 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:12:02.135131 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:02.135105 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:12:02.199305 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:02.198937 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-dzpfq" podStartSLOduration=2.037966635 podStartE2EDuration="19.198917689s" totalImagesPullingTime="17.160951054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:02.184582324 +0000 UTC m=+70.969311584" watchObservedRunningTime="2026-11-03 14:12:02.198917689 +0000 UTC m=+70.983646927" Nov 03 14:12:03.133236 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:03.132722 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-phmgc" event={"ID":"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf","Type":"ContainerStarted","Data":"d8b92bb35c5836e17c6298b1dd634960fd0c11bb45b0868b22f203e02af881c2"} Nov 03 14:12:03.177624 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:03.177537 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-phmgc" podStartSLOduration=67.946751248 podStartE2EDuration="1m12.177519082s" totalImagesPullingTime="4.230767834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:03.177393391 +0000 UTC m=+71.962122630" watchObservedRunningTime="2026-11-03 14:12:03.177519082 +0000 UTC m=+71.962248320" Nov 03 14:12:04.139375 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:04.139283 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tjsp9" event={"ID":"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09","Type":"ContainerStarted","Data":"b879f58faa4ecbe356bbad32ac9ebe8afd6adce72a5f9bc9b272aabd1d3bc481"} Nov 03 14:12:04.139911 ip-10-0-130-191 kubenswrapper[2566]: I1103 14:12:04.139546 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:12:04.358316 ip-10-0-130-191 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 14:12:04.372430 ip-10-0-130-191 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 14:12:04.372642 ip-10-0-130-191 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 14:12:04.372860 ip-10-0-130-191 systemd[1]: kubelet.service: Consumed 4.142s CPU time. Nov 03 14:12:04.386269 ip-10-0-130-191 systemd[1]: Starting Kubernetes Kubelet... Nov 03 14:12:04.508014 ip-10-0-130-191 kubenswrapper[6389]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 14:12:04.508014 ip-10-0-130-191 kubenswrapper[6389]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 14:12:04.508014 ip-10-0-130-191 kubenswrapper[6389]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 14:12:04.508014 ip-10-0-130-191 kubenswrapper[6389]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 14:12:04.510737 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510625 6389 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:12:04.510737 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510735 6389 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:12:04.510737 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510741 6389 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:12:04.510737 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510744 6389 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510748 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510752 6389 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510757 6389 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510761 6389 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510765 6389 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510769 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510773 6389 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510778 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510783 6389 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510787 6389 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510791 6389 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510796 6389 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510800 6389 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510805 6389 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510809 6389 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510813 6389 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510818 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510823 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510827 6389 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:12:04.510964 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510831 6389 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510837 6389 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510840 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510843 6389 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510845 6389 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510856 6389 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510859 6389 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510863 6389 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510868 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510872 6389 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510877 6389 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510883 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510887 6389 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510890 6389 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510894 6389 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510896 6389 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510899 6389 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510903 6389 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510905 6389 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:12:04.511647 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510908 6389 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510910 6389 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510913 6389 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510916 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510919 6389 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510921 6389 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510925 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510928 6389 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510931 6389 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510933 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510936 6389 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510938 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510941 6389 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510943 6389 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510945 6389 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510948 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510951 6389 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510953 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510956 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510958 6389 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:12:04.512212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510961 6389 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510964 6389 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510968 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510972 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510977 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510981 6389 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510984 6389 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510989 6389 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510993 6389 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.510997 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511001 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511005 6389 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511009 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511013 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511017 6389 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511022 6389 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511027 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511031 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511035 6389 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511042 6389 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:12:04.512853 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511059 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511062 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511065 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511068 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511070 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511073 6389 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511076 6389 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511078 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511083 6389 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511087 6389 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511092 6389 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511097 6389 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511100 6389 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511104 6389 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511109 6389 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511114 6389 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511118 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511121 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511125 6389 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511129 6389 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:12:04.513516 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511135 6389 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511139 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511143 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.511148 6389 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512132 6389 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512144 6389 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512149 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512154 6389 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512158 6389 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512162 6389 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512167 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512171 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512175 6389 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512178 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512180 6389 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512183 6389 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512186 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512189 6389 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512192 6389 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512195 6389 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:12:04.514248 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512198 6389 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512200 6389 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512203 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512206 6389 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512208 6389 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512211 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512213 6389 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512216 6389 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512218 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512221 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512224 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512228 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512230 6389 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512233 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512235 6389 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512238 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512241 6389 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512244 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512248 6389 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512252 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:12:04.514965 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512256 6389 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512259 6389 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512261 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512264 6389 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512267 6389 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512270 6389 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512273 6389 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512276 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512278 6389 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512281 6389 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512284 6389 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512286 6389 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512290 6389 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512294 6389 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512297 6389 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512300 6389 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512304 6389 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512308 6389 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512312 6389 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512316 6389 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:12:04.515657 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512321 6389 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512325 6389 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512329 6389 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512334 6389 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512338 6389 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512342 6389 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512344 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512347 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512349 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512352 6389 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512354 6389 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512357 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512359 6389 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512362 6389 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512364 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512367 6389 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512369 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512371 6389 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512374 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512377 6389 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:12:04.516351 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512380 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512383 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512385 6389 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512388 6389 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512392 6389 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512397 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512403 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512408 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512411 6389 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512416 6389 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512421 6389 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512425 6389 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512429 6389 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512433 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512437 6389 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512440 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512443 6389 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512447 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512451 6389 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:12:04.517138 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512455 6389 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512459 6389 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512463 6389 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512465 6389 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512468 6389 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512470 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512473 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512475 6389 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512478 6389 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512481 6389 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.512483 6389 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513238 6389 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513250 6389 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513255 6389 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513259 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513262 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513265 6389 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513269 6389 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513273 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:12:04.517784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513277 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513281 6389 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513285 6389 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513289 6389 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513293 6389 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513297 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513302 6389 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513305 6389 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513310 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513314 6389 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513318 6389 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513322 6389 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513326 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513330 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513335 6389 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513337 6389 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513340 6389 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513343 6389 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513345 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513348 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:12:04.518463 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513351 6389 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513353 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513356 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513359 6389 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513361 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513365 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513369 6389 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513374 6389 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513378 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513382 6389 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513386 6389 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513390 6389 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513394 6389 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513398 6389 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513401 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513405 6389 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513410 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513415 6389 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513419 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:12:04.519211 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513423 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513428 6389 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513433 6389 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513438 6389 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513443 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513447 6389 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513451 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513456 6389 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513460 6389 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513464 6389 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513469 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513473 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513478 6389 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513482 6389 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513492 6389 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513497 6389 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513504 6389 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513511 6389 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513517 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:12:04.519865 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513523 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513526 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513529 6389 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513531 6389 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513534 6389 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513537 6389 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513539 6389 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513542 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513545 6389 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513547 6389 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513550 6389 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513552 6389 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513555 6389 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513557 6389 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513560 6389 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513564 6389 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513567 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513569 6389 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513572 6389 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513575 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:12:04.520489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513580 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513584 6389 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513588 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513593 6389 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513597 6389 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513602 6389 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513623 6389 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513630 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513634 6389 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513638 6389 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513642 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513646 6389 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513651 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513658 6389 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513663 6389 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513667 6389 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513671 6389 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513675 6389 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513680 6389 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.513683 6389 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513824 6389 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 14:12:04.521191 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513848 6389 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513857 6389 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513862 6389 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513867 6389 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513870 6389 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513874 6389 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513883 6389 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513888 6389 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513893 6389 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513899 6389 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513903 6389 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513909 6389 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513914 6389 flags.go:64] FLAG: --cgroup-root="" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513918 6389 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513923 6389 flags.go:64] FLAG: --client-ca-file="" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513927 6389 flags.go:64] FLAG: --cloud-provider="external" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513933 6389 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513939 6389 flags.go:64] FLAG: --cluster-domain="" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513944 6389 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513947 6389 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513950 6389 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513954 6389 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513961 6389 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 14:12:04.521944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513965 6389 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513970 6389 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513978 6389 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513982 6389 flags.go:64] FLAG: --contention-profiling="false" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513986 6389 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513990 6389 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.513995 6389 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514000 6389 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514008 6389 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514013 6389 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514017 6389 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514022 6389 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514027 6389 flags.go:64] FLAG: --enable-server="true" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514032 6389 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514039 6389 flags.go:64] FLAG: --event-burst="100" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514046 6389 flags.go:64] FLAG: --event-qps="50" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514049 6389 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514052 6389 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514055 6389 flags.go:64] FLAG: --eviction-hard="" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514059 6389 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514063 6389 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514066 6389 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514069 6389 flags.go:64] FLAG: --eviction-soft="" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514072 6389 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514075 6389 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 14:12:04.522820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514078 6389 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514081 6389 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514085 6389 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514088 6389 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514091 6389 flags.go:64] FLAG: --feature-gates="" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514095 6389 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514098 6389 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514103 6389 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514108 6389 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514113 6389 flags.go:64] FLAG: --healthz-port="10248" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514119 6389 flags.go:64] FLAG: --help="false" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514124 6389 flags.go:64] FLAG: --hostname-override="ip-10-0-130-191.ec2.internal" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514129 6389 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514133 6389 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514138 6389 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514142 6389 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514146 6389 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514149 6389 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514153 6389 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514158 6389 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514162 6389 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514167 6389 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514175 6389 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514180 6389 flags.go:64] FLAG: --kube-reserved="" Nov 03 14:12:04.523761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514185 6389 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514190 6389 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514195 6389 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514200 6389 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514204 6389 flags.go:64] FLAG: --lock-file="" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514209 6389 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514213 6389 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514218 6389 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514229 6389 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514232 6389 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514236 6389 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514239 6389 flags.go:64] FLAG: --logging-format="text" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514242 6389 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514245 6389 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514248 6389 flags.go:64] FLAG: --manifest-url="" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514251 6389 flags.go:64] FLAG: --manifest-url-header="" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514257 6389 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514260 6389 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514264 6389 flags.go:64] FLAG: --max-pods="110" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514267 6389 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514271 6389 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514274 6389 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514276 6389 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514279 6389 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514283 6389 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 14:12:04.524646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514288 6389 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514300 6389 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514305 6389 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514309 6389 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514312 6389 flags.go:64] FLAG: --pod-cidr="" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514316 6389 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514319 6389 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514322 6389 flags.go:64] FLAG: --pods-per-core="0" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514324 6389 flags.go:64] FLAG: --port="10250" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514327 6389 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514330 6389 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bcaa5720b29ec280" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514334 6389 flags.go:64] FLAG: --qos-reserved="" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514338 6389 flags.go:64] FLAG: --read-only-port="10255" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514344 6389 flags.go:64] FLAG: --register-node="true" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514349 6389 flags.go:64] FLAG: --register-with-taints="" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514353 6389 flags.go:64] FLAG: --registry-burst="10" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514356 6389 flags.go:64] FLAG: --registry-qps="5" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514363 6389 flags.go:64] FLAG: --reserved-cpus="" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514368 6389 flags.go:64] FLAG: --reserved-memory="" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514374 6389 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514377 6389 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514380 6389 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514383 6389 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514386 6389 flags.go:64] FLAG: --runonce="false" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514389 6389 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514392 6389 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514395 6389 flags.go:64] FLAG: --seccomp-default="false" Nov 03 14:12:04.525506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514399 6389 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514402 6389 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514405 6389 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514408 6389 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514411 6389 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514414 6389 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514417 6389 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514420 6389 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514423 6389 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514426 6389 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514430 6389 flags.go:64] FLAG: --system-cgroups="" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514434 6389 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514443 6389 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514447 6389 flags.go:64] FLAG: --tls-cert-file="" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514452 6389 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514458 6389 flags.go:64] FLAG: --tls-min-version="" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514464 6389 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514467 6389 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514470 6389 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514473 6389 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514476 6389 flags.go:64] FLAG: --v="2" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514482 6389 flags.go:64] FLAG: --version="false" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514490 6389 flags.go:64] FLAG: --vmodule="" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514496 6389 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.514502 6389 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 14:12:04.526564 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514698 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514706 6389 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514711 6389 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514715 6389 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514718 6389 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514720 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514723 6389 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514725 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514729 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514733 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514737 6389 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514742 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514746 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514750 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514756 6389 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514764 6389 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514769 6389 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514777 6389 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514781 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514784 6389 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:12:04.527784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514788 6389 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514793 6389 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514797 6389 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514801 6389 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514805 6389 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514810 6389 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514814 6389 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514819 6389 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514822 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514827 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514831 6389 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514836 6389 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514840 6389 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514844 6389 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514849 6389 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514853 6389 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514858 6389 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514863 6389 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514868 6389 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:12:04.528555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514872 6389 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514877 6389 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514880 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514883 6389 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514886 6389 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514889 6389 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514891 6389 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514894 6389 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514897 6389 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514899 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514903 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514905 6389 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514908 6389 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514910 6389 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514913 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514916 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514918 6389 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514921 6389 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514923 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514926 6389 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:12:04.529471 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514930 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514934 6389 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514940 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514944 6389 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514949 6389 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514953 6389 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514958 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514962 6389 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514966 6389 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514970 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514974 6389 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514979 6389 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514984 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514988 6389 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514992 6389 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.514998 6389 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515002 6389 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515007 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515011 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:12:04.530187 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515015 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515019 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515023 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515028 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515030 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515033 6389 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515035 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515040 6389 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515044 6389 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515056 6389 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515060 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515065 6389 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515070 6389 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515074 6389 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515079 6389 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515083 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515086 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515089 6389 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515092 6389 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:12:04.530904 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515094 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515097 6389 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515099 6389 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515102 6389 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515106 6389 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515111 6389 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515115 6389 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515120 6389 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515124 6389 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.515169 6389 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515215 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515224 6389 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515229 6389 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515233 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515238 6389 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515242 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:12:04.531633 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515247 6389 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515251 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515255 6389 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515259 6389 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515263 6389 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515268 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515272 6389 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515276 6389 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515279 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515281 6389 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515284 6389 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515287 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515289 6389 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515294 6389 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515297 6389 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515299 6389 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515302 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515304 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515307 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:12:04.532194 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515309 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515312 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515315 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515319 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515324 6389 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515328 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515335 6389 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515342 6389 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515347 6389 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515350 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515353 6389 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515355 6389 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515358 6389 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515361 6389 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515365 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515370 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515374 6389 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515378 6389 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515382 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:12:04.532929 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515387 6389 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515391 6389 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515395 6389 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515400 6389 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515405 6389 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515409 6389 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515414 6389 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515421 6389 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515427 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515432 6389 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515436 6389 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515440 6389 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515445 6389 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515449 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515455 6389 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515460 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515463 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515465 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515470 6389 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515474 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:12:04.533632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515478 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515482 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515489 6389 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515496 6389 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515501 6389 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515504 6389 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515506 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515509 6389 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515512 6389 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515516 6389 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515520 6389 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515524 6389 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515528 6389 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515532 6389 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515536 6389 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515540 6389 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515544 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515548 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515553 6389 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:12:04.534222 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515559 6389 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515564 6389 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515568 6389 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515572 6389 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515577 6389 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515581 6389 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515584 6389 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515589 6389 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515593 6389 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515598 6389 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515602 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515622 6389 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515626 6389 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515631 6389 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515635 6389 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515640 6389 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515645 6389 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515649 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515651 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515657 6389 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:12:04.534927 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515660 6389 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515663 6389 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.515667 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.520499 6389 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.520513 6389 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.520538 6389 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.520545 6389 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520639 6389 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520647 6389 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520652 6389 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520657 6389 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520662 6389 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520666 6389 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520670 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520675 6389 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520679 6389 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520683 6389 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520687 6389 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520692 6389 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:12:04.535575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520696 6389 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520703 6389 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520709 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520715 6389 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520719 6389 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520724 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520729 6389 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520733 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520736 6389 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520741 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520743 6389 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520746 6389 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520749 6389 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520751 6389 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520756 6389 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520759 6389 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520762 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520765 6389 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520768 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:12:04.536308 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520770 6389 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520773 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520776 6389 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520778 6389 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520781 6389 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520783 6389 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520786 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520788 6389 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520792 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520794 6389 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520797 6389 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520800 6389 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520802 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520805 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520807 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520810 6389 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520812 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520815 6389 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520817 6389 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520820 6389 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:12:04.536963 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520822 6389 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520825 6389 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520827 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520830 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520833 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520835 6389 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520838 6389 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520841 6389 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520843 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520846 6389 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520848 6389 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520852 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520856 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520861 6389 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520866 6389 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520870 6389 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520874 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520878 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520883 6389 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520887 6389 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:12:04.537646 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520891 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520895 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520899 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520902 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520904 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520907 6389 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520910 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520914 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520918 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520923 6389 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520925 6389 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520928 6389 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520931 6389 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520933 6389 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520936 6389 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520938 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520942 6389 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520944 6389 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520947 6389 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520950 6389 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:12:04.538345 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520952 6389 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520955 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520958 6389 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520960 6389 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520962 6389 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520965 6389 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520967 6389 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520969 6389 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520972 6389 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520976 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520981 6389 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520986 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520991 6389 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520995 6389 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.520999 6389 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.521037 6389 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 14:12:04.539074 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521200 6389 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521207 6389 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521211 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521216 6389 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521220 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521225 6389 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521230 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521234 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521238 6389 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521243 6389 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521248 6389 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521252 6389 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521255 6389 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521259 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521261 6389 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521264 6389 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521267 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521269 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521272 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:12:04.539643 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521275 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521279 6389 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521283 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521287 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521292 6389 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521296 6389 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521299 6389 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521303 6389 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521307 6389 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521311 6389 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521316 6389 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521320 6389 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521324 6389 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521329 6389 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521333 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521337 6389 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521341 6389 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521345 6389 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521350 6389 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521354 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:12:04.540257 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521358 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521361 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521364 6389 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521368 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521372 6389 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521377 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521382 6389 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521387 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521391 6389 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521396 6389 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521399 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521403 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521405 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521409 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521413 6389 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521418 6389 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521423 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521427 6389 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521432 6389 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:12:04.540899 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521436 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521441 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521445 6389 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521449 6389 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521454 6389 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521458 6389 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521464 6389 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521468 6389 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521472 6389 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521474 6389 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521477 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521480 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521482 6389 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521485 6389 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521487 6389 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521490 6389 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521492 6389 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521495 6389 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521497 6389 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:12:04.541507 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521500 6389 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521502 6389 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521506 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521508 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521511 6389 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521514 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521516 6389 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521519 6389 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521521 6389 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521524 6389 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521526 6389 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521529 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521532 6389 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521536 6389 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521539 6389 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521542 6389 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521545 6389 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521547 6389 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521550 6389 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:12:04.542269 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521552 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521555 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521557 6389 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521560 6389 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521562 6389 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521564 6389 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521567 6389 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521569 6389 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521573 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521578 6389 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.521630 6389 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521683 6389 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521690 6389 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521694 6389 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521696 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521702 6389 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:12:04.542934 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521706 6389 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521708 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521711 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521714 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521717 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521719 6389 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521722 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521724 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521726 6389 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521729 6389 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521731 6389 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521733 6389 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521736 6389 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521738 6389 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521741 6389 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521743 6389 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521746 6389 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521748 6389 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521751 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521754 6389 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:12:04.543484 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521757 6389 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521761 6389 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521766 6389 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521769 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521773 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521778 6389 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521782 6389 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521786 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521790 6389 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521794 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521799 6389 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521805 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521809 6389 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521815 6389 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521820 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521824 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521829 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521833 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521837 6389 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521841 6389 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:12:04.544209 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521845 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521849 6389 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521852 6389 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521854 6389 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521858 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521860 6389 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521863 6389 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521865 6389 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521867 6389 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521871 6389 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521877 6389 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521880 6389 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521883 6389 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521886 6389 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521888 6389 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521890 6389 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521894 6389 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521897 6389 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521902 6389 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521908 6389 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:12:04.544908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521912 6389 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521916 6389 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521919 6389 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521923 6389 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521926 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521928 6389 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521932 6389 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521935 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521938 6389 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521942 6389 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521945 6389 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521949 6389 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521955 6389 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521960 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521964 6389 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521970 6389 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521973 6389 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521976 6389 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521979 6389 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:12:04.545584 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521982 6389 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521984 6389 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521987 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521989 6389 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521992 6389 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521995 6389 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.521997 6389 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522000 6389 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522002 6389 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522005 6389 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522007 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522009 6389 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522013 6389 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522015 6389 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522018 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522021 6389 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522024 6389 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522027 6389 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522029 6389 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522032 6389 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:12:04.546349 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522035 6389 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:04.522040 6389 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.522097 6389 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.523352 6389 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.523424 6389 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.523667 6389 server.go:1014] "Starting client certificate rotation" Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.523807 6389 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.523899 6389 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 14:05:51 +0000 UTC" deadline="2028-05-17 04:10:05.670987798 +0000 UTC" Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.523936 6389 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13453h58m1.147054381s" Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.524300 6389 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.525752 6389 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.529768 6389 log.go:25] "Validated CRI v1 runtime API" Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.532053 6389 log.go:25] "Validated CRI v1 image API" Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.532942 6389 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 14:12:04.547158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.535406 6389 fs.go:135] Filesystem UUIDs: map[564230ad-cbaa-4291-a44a-18780785af32:/dev/nvme0n1p3 6db23011-393a-4b9f-86b9-83631f4fa445:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 14:12:04.547868 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.535428 6389 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03341514a396ff5f2bd42621fbf5e52b36887ce841edb6497040a8fa71e68566/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03341514a396ff5f2bd42621fbf5e52b36887ce841edb6497040a8fa71e68566/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03a2ab831fb7262165c5cdc198b059eb0636bc1c1e20be57ffb5bda465174c8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03a2ab831fb7262165c5cdc198b059eb0636bc1c1e20be57ffb5bda465174c8a/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/040234a2b252b62d917e66263b509fcf4e1f7886f408e65d77e776a2dbff2fc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/040234a2b252b62d917e66263b509fcf4e1f7886f408e65d77e776a2dbff2fc9/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0483756c8af80f00eb23edb5bc09c749cfc1dd101501f56ccd30bf6498439cc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0483756c8af80f00eb23edb5bc09c749cfc1dd101501f56ccd30bf6498439cc6/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a431ebd473676f2458785a54cd60c6116d7c401339a8f5af2cb6dbbdac4973e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a431ebd473676f2458785a54cd60c6116d7c401339a8f5af2cb6dbbdac4973e/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b32bff675427e2f8532946e3b2d9082bd1900a0c52210171e9a90ec60c5265b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b32bff675427e2f8532946e3b2d9082bd1900a0c52210171e9a90ec60c5265b/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2cd11b588eab8bb8eb0a93c647495527199429a62ff928e732585039a780f536/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2cd11b588eab8bb8eb0a93c647495527199429a62ff928e732585039a780f536/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4083a692b9eb0f2a859ddcc5f55cd9a3aff7c5aaa4d334def8e0f2df046c5530/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4083a692b9eb0f2a859ddcc5f55cd9a3aff7c5aaa4d334def8e0f2df046c5530/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65ffbf449cc14e3ad047a0449e575b697995b6fcfc4f8f8c73374c57c94918fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65ffbf449cc14e3ad047a0449e575b697995b6fcfc4f8f8c73374c57c94918fa/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/695791e9fed3c00fe43512d295793f792f61086f14c4ae256b88615f2c24e9b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/695791e9fed3c00fe43512d295793f792f61086f14c4ae256b88615f2c24e9b9/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69fc2d0a5d65d549a0169817881ddf3a96f96b3096dadd736528ef36dfbfe900/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69fc2d0a5d65d549a0169817881ddf3a96f96b3096dadd736528ef36dfbfe900/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f2ab086794d480503a53b41b8a854f866fe036b494e8644cc5bf586e1cdbb37/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f2ab086794d480503a53b41b8a854f866fe036b494e8644cc5bf586e1cdbb37/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71026b6e406cc80284587c4bfa21304cca3450f73ff547b89b9b085d4d4a2898/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71026b6e406cc80284587c4bfa21304cca3450f73ff547b89b9b085d4d4a2898/userdata/shm major:0 minor:318 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/999b1867c7515b06d1e6db3f17fe4b82cd262b99539615faabe4170a1711fd9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/999b1867c7515b06d1e6db3f17fe4b82cd262b99539615faabe4170a1711fd9b/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aff3c5989d7f586de304608ba8eaced771e012ed3f4770980b7aeb7b8d3fa9ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aff3c5989d7f586de304608ba8eaced771e012ed3f4770980b7aeb7b8d3fa9ee/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb50e172753912610fb1cffdee52e795180f08135ce69cc8d4f92cdbd470dc70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb50e172753912610fb1cffdee52e795180f08135ce69cc8d4f92cdbd470dc70/userdata/shm major:0 minor:363 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0b128e2f7a5c2e0f1faac54a5a1a33b7aab3700813da44785a9d1d0fe406f2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0b128e2f7a5c2e0f1faac54a5a1a33b7aab3700813da44785a9d1d0fe406f2e/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d762ba31bc07b465c2882275574c371615a22b61191d8c4c8ff5bc614c715e6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d762ba31bc07b465c2882275574c371615a22b61191d8c4c8ff5bc614c715e6d/userdata/shm major:0 minor:99 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9a77ad916f5ee90d19fd297849f1a6e4f9be2ee781e2be5d061d74572af8362/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9a77ad916f5ee90d19fd297849f1a6e4f9be2ee781e2be5d061d74572af8362/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0d7a728bd64603d00f7123064da977fb3e58c376216ddea8902aa82ba66a3b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0d7a728bd64603d00f7123064da977fb3e58c376216ddea8902aa82ba66a3b0/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0e2dd814d101ad57d106df1be9fca646bfeafb1c40f40b76f8f91a0eb953367/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0e2dd814d101ad57d106df1be9fca646bfeafb1c40f40b76f8f91a0eb953367/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6282608a6482faf891a0cf254501636a13b0138457ce64ca15f934ebc1710f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6282608a6482faf891a0cf254501636a13b0138457ce64ca15f934ebc1710f0/userdata/shm major:0 minor:361 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fafcf58717120bce6dc686e7b602493ee4ad8879d233785ca2b2bf90c909e84e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fafcf58717120bce6dc686e7b602493ee4ad8879d233785ca2b2bf90c909e84e/userdata/shm major:0 minor:385 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05a67278-34b4-462d-b4cb-9250800bf3db/volumes/kubernetes.io~projected/kube-api-access-wkjnc:{mountpoint:/var/lib/kubelet/pods/05a67278-34b4-462d-b4cb-9250800bf3db/volumes/kubernetes.io~projected/kube-api-access-wkjnc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0694b8ee-14f6-495d-a662-8f06bdfe1c66/volumes/kubernetes.io~projected/kube-api-access-m7d5k:{mountpoint:/var/lib/kubelet/pods/0694b8ee-14f6-495d-a662-8f06bdfe1c66/volumes/kubernetes.io~projected/kube-api-access-m7d5k major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40e20104-e97c-41b2-8530-abb2dec148a2/volumes/kubernetes.io~projected/kube-api-access-kq4pb:{mountpoint:/var/lib/kubelet/pods/40e20104-e97c-41b2-8530-abb2dec148a2/volumes/kubernetes.io~projected/kube-api-access-kq4pb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~projected/kube-api-access-tjxcm:{mountpoint:/var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~projected/kube-api-access-tjxcm major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61d8c282-03cc-4a20-a999-6cda0b14c87e/volumes/kubernetes.io~projected/kube-api-access-7j5qj:{mountpoint:/var/lib/kubelet/pods/61d8c282-03cc-4a20-a999-6cda0b14c87e/volumes/kubernetes.io~projected/kube-api-access-7j5qj major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61d8c282-03cc-4a20-a999-6cda0b14c87e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/61d8c282-03cc-4a20-a999-6cda0b14c87e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61d8c282-03cc-4a20-a999-6cda0b14c87e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/61d8c282-03cc-4a20-a999-6cda0b14c87e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf/volumes/kubernetes.io~projected/kube-api-access-dch9b:{mountpoint:/var/lib/kubelet/pods/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf/volumes/kubernetes.io~projected/kube-api-access-dch9b major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e380172-2ed6-4bd3-9fdb-83a694b400d0/volumes/kubernetes.io~projected/kube-api-access-jhgqh:{mountpoint:/var/lib/kubelet/pods/8e380172-2ed6-4bd3-9fdb-83a694b400d0/volumes/kubernetes.io~projected/kube-api-access-jhgqh major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e380172-2ed6-4bd3-9fdb-83a694b400d0/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8e380172-2ed6-4bd3-9fdb-83a694b400d0/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e380172-2ed6-4bd3-9fdb-83a694b400d0/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/8e380172-2ed6-4bd3-9fdb-83a694b400d0/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/905b521a-2b6a-4fb6-ba5b-33311a2613b0/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/905b521a-2b6a-4fb6-ba5b-33311a2613b0/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/977c941d-36f2-4c16-8396-95bdce627a74/volumes/kubernetes.io~projected/kube-api-access-g2q54:{mountpoint:/var/lib/kubelet/pods/977c941d-36f2-4c16-8396-95bdce627a74/volumes/kubernetes.io~projected/kube-api-access-g2q54 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/977c941d-36f2-4c16-8396-95bdce627a74/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/977c941d-36f2-4c16-8396-95bdce627a74/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a22219af-1262-42a9-8cbe-8e6323e4375b/volumes/kubernetes.io~projected/kube-api-access-wd8sm:{mountpoint:/var/lib/kubelet/pods/a22219af-1262-42a9-8cbe-8e6323e4375b/volumes/kubernetes.io~projected/kube-api-access-wd8sm major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a22219af-1262-42a9-8cbe-8e6323e4375b/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a22219af-1262-42a9-8cbe-8e6323e4375b/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a22219af-1262-42a9-8cbe-8e6323e4375b/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a22219af-1262-42a9-8cbe-8e6323e4375b/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56/volumes/kubernetes.io~projected/kube-api-access-47v7j:{mountpoint:/var/lib/kubelet/pods/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56/volumes/kubernetes.io~projected/kube-api-access-47v7j major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b00819ff-6b3d-4e36-995b-2d671e0dc9f6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b00819ff-6b3d-4e36-995b-2d671e0dc9f6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b00819ff-6b3d-4e36-995b-2d671e0dc9f6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b00819ff-6b3d-4e36-995b-2d671e0dc9f6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b00819ff-6b3d-4e36-995b-2d671e0dc9f6/volumes/kubernetes.io~projected/kube-api-access-5dlv2:{mountpoint:/var/lib/kubelet/pods/b00819ff-6b3d-4e36-995b-2d671e0dc9f6/volumes/kubernetes.io~projected/kube-api-access-5dlv2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea/volumes/kubernetes.io~projected/kube-api-access-f2wbc:{mountpoint:/var/lib/kubelet/pods/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea/volumes/kubernetes.io~projected/kube-api-access-f2wbc major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b586d272-c9c0-455f-aec9-333ae6d0c69a/volumes/kubernetes.io~projected/kube-api-access-7d4lt:{mountpoint:/var/lib/kubelet/pods/b586d272-c9c0-455f-aec9-333ae6d0c69a/volumes/kubernetes.io~projected/kube-api-access-7d4lt major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d153593a-c25f-4bef-acca-ea7e50b32c87/volumes/kubernetes.io~projected/kube-api-access-z2mvt:{mountpoint:/var/lib/kubelet/pods/d153593a-c25f-4bef-acca-ea7e50b32c87/volumes/kubernetes.io~projected/kube-api-access-z2mvt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d/volumes/kubernetes.io~projected/kube-api-access-xkjjv:{mountpoint:/var/lib/kubelet/pods/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d/volumes/kubernetes.io~projected/kube-api-access-xkjjv major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dad8ff83-33fe-4482-927e-8b70b1e72643/volumes/kubernetes.io~projected/kube-api-access-k45tm:{mountpoint:/var/lib/kubelet/pods/dad8ff83-33fe-4482-927e-8b70b1e72643/volumes/kubernetes.io~projected/kube-api-access-k45tm major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dad8ff83-33fe-4482-927e-8b70b1e72643/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/dad8ff83-33fe-4482-927e-8b70b1e72643/volumes/kubernetes.io~secret/metrics-certs major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edb1d4a1-0656-4417-8b7d-86a68db196ee/volumes/kubernetes.io~projected/kube-api-access-8mw4d:{mountpoint:/var/lib/kubelet/pods/edb1d4a1-0656-4417-8b7d-86a68db196ee/volumes/kubernetes.io~projected/kube-api-access-8mw4d major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc/volumes/kubernetes.io~projected/kube-api-access-4k6zn:{mountpoint:/var/lib/kubelet/pods/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc/volumes/kubernetes.io~projected/kube-api-access-4k6zn major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f394a27c-5456-4b99-8b59-b89d25aae03b/volumes/kubernetes.io~projected/kube-api-access-xpp9t:{mountpoint:/var/lib/kubelet/pods/f394a27c-5456-4b99-8b59-b89d25aae03b/volumes/kubernetes.io~projected/kube-api-access-xpp9t major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f394a27c-5456-4b99-8b59-b89d25aae03b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f394a27c-5456-4b99-8b59-b89d25aae03b/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5/volumes/kubernetes.io~projected/kube-api-access-h4z4z:{mountpoint:/var/lib/kubelet/pods/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5/volumes/kubernetes.io~projected/kube-api-access-h4z4z major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:357 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a80b0e89d1f32020078bac95b50abff5d04f1709d7a022b005347072a746db3c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6c3624f2c25fb5826074efcf7295ad29751dd92c3b30745aff078c3506af95ec/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/90b542e95e6e791b1572564e778e7d2eb861499590f796c78eeb0945bba68b56/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/26ba959ec18216810a3d0ab8dbbc2b5c8ffb33f1332889845c58519886ee31e1/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/49330d460dd6c42619e62940ae2ba747e77b0fe3bbb63e3d6bd481846971e6b9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d0f1cc898bf020f8a028cfffb3e913f748013ea573f8517494debfa426a34822/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5b35572903b38a351b2235b2acc7ec0330c80cc7dfc6705a234449a18f31e030/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4a647d694ce71ca3d1cdd5bfe5c06a7150859fc200bb365140926c3cb42536be/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f60d407b09e83834089e9ac237d37cd7f65a1c46c741188d6e25ee2edf3f4a51/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/cb7ad1e7223fff86b7deabadb53e650c74238d487ac21ae9892efa5ff338c337/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/7ff6c98ed5f2560c4169bc653ded4e8e489455b0b0f073b5faabaa28b029211b/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/a145efadd2a95a220d924dcc6842f4fb22ce4eec16592c0b40fd0d57be41fc33/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/545a283a4eca5d00b0f23403ce94affa8864195d616f4f8cbcdfda46ebd356dd/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/945b948c60ab26de695518a195ee2e574ccfbcfa90360de38c42bd6c2ad6bb90/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a1807d259a0d7e101719e28a963dc9741a54fbb7772ee9ce0646ff973cd25f38/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7e28e369d48c6c195eef20e2a5c73efe2b85aeb3a111b52896679f212dc15e9a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/0baa9e0fe652605f1b1c42793e12f0d2971acd7a13428f6638bb4402e7677ed1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5b5a37851a46182210565a7a6ab66b67938b4ac381052e916cdc2741955061ba/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b4392fefcf8a846e50b13fd359689747029baeff385cf42dec6068b29b5b32a6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cf390b63e0ff6425afdb229d928e86b40a83bd2ae2e2275f98ded969d5b9f763/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2d7a1584de4069a744c455ffc3591daa67e7aac2c3cb060207d18eb8a245671c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/b82656fdfa2b3ca0587a8fa6924d664e1f74a0c2839cbaba8e12b661e8977b5d/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/e6934d5fcc46eca6995ac047fbb57b99e13784846d1307aefc5db27031dfaf35/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/faae9b4ba13fa7a920480bc463705ac48cb48b6f3bbba41e9cef593806263056/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/1b210bf4b71f8a13f1f8d72ccc3fb90b7ecd96fa830d55243a25d661019e6171/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/bc53508f317642ddc4ea06c5b6f408f48392ccd46e5de39b742f4d31f2769c58/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/82952a4b9c0f6cd11f61c1620b6c5f972d496704bb9127c57241a1d864e5504e/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/9ffd1ab826731115f8c14a6e8524683b19ff02fbd1c461f47195b9d34105b191/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/045160812ca2fb60ceb8400665f315be77db862ea1fa1323e07aadc182a69224/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/48873e85515a61e3882509bd9fb5c55ce89e837e1798978a8c48eacfe9be66fe/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/879185a1888bb94b043e3ededf1826eaebf5305b51744dd7207cfac9e47f312c/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/0e9a6d04a11856bb7f2963584cfab83ab05e4992c28934abef852b120cb8311f/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/e4fd0e312ca8fe43a579473e1488eccb42301ab1ff1a4412ec9ba2210dbd0afa/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/425ee76b0426a57794f87971de17f2438d216ec3203be061f910449df69dffcd/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/e55a37d26a8f7ed8792ccc821f77a54783c57c26923ceb7fb0abf5b895b85714/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/06a77409c451de98c4483742cb012b9f270945a6293c04b8a6f42cb3cbe3045a/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/da314c8177d894efa29720a3e753675fb9a88995bac125691310dba6f4849631/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/8c826032a37daba80745a07cb6a974812636fa16f9c70bc5d4584890a8df6f2a/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/d95c25bb380baf93c8ee2606efae328589eadbd7a39e6fb27e1ecdf57f79f44a/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/34c2d92ae7dba9fd83c0cc8e4c7b116bed3cba753e72abf5dc5e7353a2ca0974/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/1030eb0e16f796aed7c27df752349b569f13096384d7dc8f6edfb73b8fe46764/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/fa9217fb45d0d0180df614c6f187692dbce4d1c72ae0ea2f5c2c3e62bc5bae14/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/c2c1c61f84c9219b577086e40839ce712be41e6405a5a9ec4c1216618ee004e0/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/a5669804d93b24a519a429823269a3aeb7cb268f6b939550fb9932536893cbee/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/971ebfcbd651cb1d8f53debd105f9252fdda3393db10b67beb3f0c1000e23dfd/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/93444ebff386ea02c62e7f66580d34f0b8b0bcc396a49c73ff6fd871177e3642/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/45cba05efd50205444fc20f0194fa98c3e8e561b3174997cf146390794cb1fa3/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/f246ae048f51468095361b09a49eea890a7f2f2dfbcb3ff73e3650dff8a6743d/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/4851076f5687c3e2e9fec70f87dfc94558d2ab22770e3ff2464dc65f2c308ec2/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/5bd7943c6e8b809275badc654d2167b5b4fa39894b490bb06d103feac5bca031/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/7ef1089f166c5cf78f3672507312a9ecb73d609c0211836de0cb2b9887f16149/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/db749c48fc842b1406549ede8a29002094d012505766102c7c52ed09214b15ac/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/8530215e7198deb5a1ea0eb84bbbd357ae86ca583b98c2b1b22c6b02243838a9/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/f822b549b65e1c7501cfb117215b92455442f90fe7f5f63f395fbc2592684442/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/a4e55195f7b4928731ba19a9237187254643085dddfd5613e3c8df4fd6b73d71/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/05f871913b99af00b295a6808556fd8db2da714a482de19e313ab6d07241ecba/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/abbc05c44bb41eda5d5d5bf10427e1d2ee4e30d08412d83af83bc91d00f0b1f3/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/d0898020d4d71b9db739d178a2d03c70c54fedcb6dbec1f96b38d49fba96d135/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/2d3e4f60e580fe0b98157f6d6029f9bc870159a44fe25fbd34e594a08e12205f/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/22b44fb07a4cb527a88e132b4b1ca2cc24fd91314c21cdcf5d055b1f3ab6348a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f665b3196407172d4be59e9ffa23eda8d215b86c2f89dbada6910d6197939684/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/29700e7c061c21abab287c1c1fb67cc7f89af966130106e12109e9206598af69/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/38490d959d65ce655fb98ca72bd7f31422a29767776b525df949842f25fdb4ad/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c45d92a72176adf23557ed7cd7b17f98bca443dd85047206f97ef5950e333bc4/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/aeb0f316393fc5634196af9fc06366c5edd79b148b3d6bbb372c3a21b38d6ac3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/4b699f8fb325917df2c40711985f99252938fe9ea557880b454006bbf5107fdc/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/7433285402a530b6fb15a19b8b6a7464f95cceb0b418d379d7e05d1d1da2d8ac/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/b44f5fccb83e377253d0c007a0695d3703ec8306c8fdd634ece555c039c2c0b0/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.544522 6389 manager.go:217] Machine: {Timestamp:2026-11-03 14:12:04.543896077 +0000 UTC m=+0.102532557 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100190 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21bbbd861ef3145e25fe3ecbb4855c SystemUUID:ec21bbbd-861e-f314-5e25-fe3ecbb4855c BootID:fb6faccc-3d96-4c98-a158-474bef4f7817 Filesystems:[{Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2cd11b588eab8bb8eb0a93c647495527199429a62ff928e732585039a780f536/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:314 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/977c941d-36f2-4c16-8396-95bdce627a74/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/8e380172-2ed6-4bd3-9fdb-83a694b400d0/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:310 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fafcf58717120bce6dc686e7b602493ee4ad8879d233785ca2b2bf90c909e84e/userdata/shm DeviceMajor:0 DeviceMinor:385 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0d7a728bd64603d00f7123064da977fb3e58c376216ddea8902aa82ba66a3b0/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/695791e9fed3c00fe43512d295793f792f61086f14c4ae256b88615f2c24e9b9/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9a77ad916f5ee90d19fd297849f1a6e4f9be2ee781e2be5d061d74572af8362/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f2ab086794d480503a53b41b8a854f866fe036b494e8644cc5bf586e1cdbb37/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0e2dd814d101ad57d106df1be9fca646bfeafb1c40f40b76f8f91a0eb953367/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~projected/kube-api-access-tjxcm DeviceMajor:0 DeviceMinor:317 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea/volumes/kubernetes.io~projected/kube-api-access-f2wbc DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a22219af-1262-42a9-8cbe-8e6323e4375b/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:383 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/05a67278-34b4-462d-b4cb-9250800bf3db/volumes/kubernetes.io~projected/kube-api-access-wkjnc DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f394a27c-5456-4b99-8b59-b89d25aae03b/volumes/kubernetes.io~projected/kube-api-access-xpp9t DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/61d8c282-03cc-4a20-a999-6cda0b14c87e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:344 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:316 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03a2ab831fb7262165c5cdc198b059eb0636bc1c1e20be57ffb5bda465174c8a/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b00819ff-6b3d-4e36-995b-2d671e0dc9f6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6282608a6482faf891a0cf254501636a13b0138457ce64ca15f934ebc1710f0/userdata/shm DeviceMajor:0 DeviceMinor:361 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc/volumes/kubernetes.io~projected/kube-api-access-4k6zn DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e380172-2ed6-4bd3-9fdb-83a694b400d0/volumes/kubernetes.io~projected/kube-api-access-jhgqh DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0483756c8af80f00eb23edb5bc09c749cfc1dd101501f56ccd30bf6498439cc6/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/040234a2b252b62d917e66263b509fcf4e1f7886f408e65d77e776a2dbff2fc9/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a22219af-1262-42a9-8cbe-8e6323e4375b/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:379 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dad8ff83-33fe-4482-927e-8b70b1e72643/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:352 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:234 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65ffbf449cc14e3ad047a0449e575b697995b6fcfc4f8f8c73374c57c94918fa/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/61d8c282-03cc-4a20-a999-6cda0b14c87e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:343 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/608de1a1-c806-48bb-aecc-def971eca078/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:315 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69fc2d0a5d65d549a0169817881ddf3a96f96b3096dadd736528ef36dfbfe900/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4083a692b9eb0f2a859ddcc5f55cd9a3aff7c5aaa4d334def8e0f2df046c5530/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b32bff675427e2f8532946e3b2d9082bd1900a0c52210171e9a90ec60c5265b/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f394a27c-5456-4b99-8b59-b89d25aae03b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/977c941d-36f2-4c16-8396-95bdce627a74/volumes/kubernetes.io~projected/kube-api-access-g2q54 DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56/volumes/kubernetes.io~projected/kube-api-access-47v7j DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0694b8ee-14f6-495d-a662-8f06bdfe1c66/volumes/kubernetes.io~projected/kube-api-access-m7d5k DeviceMajor:0 DeviceMinor:335 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b586d272-c9c0-455f-aec9-333ae6d0c69a/volumes/kubernetes.io~projected/kube-api-access-7d4lt DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e380172-2ed6-4bd3-9fdb-83a694b400d0/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71026b6e406cc80284587c4bfa21304cca3450f73ff547b89b9b085d4d4a2898/userdata/shm DeviceMajor:0 DeviceMinor:318 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb50e172753912610fb1cffdee52e795180f08135ce69cc8d4f92cdbd470dc70/userdata/shm DeviceMajor:0 DeviceMinor:363 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/999b1867c7515b06d1e6db3f17fe4b82cd262b99539615faabe4170a1711fd9b/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:357 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/b00819ff-6b3d-4e36-995b-2d671e0dc9f6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40e20104-e97c-41b2-8530-abb2dec148a2/volumes/kubernetes.io~projected/kube-api-access-kq4pb DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d153593a-c25f-4bef-acca-ea7e50b32c87/volumes/kubernetes.io~projected/kube-api-access-z2mvt DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0b128e2f7a5c2e0f1faac54a5a1a33b7aab3700813da44785a9d1d0fe406f2e/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:232 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/905b521a-2b6a-4fb6-ba5b-33311a2613b0/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03341514a396ff5f2bd42621fbf5e52b36887ce841edb6497040a8fa71e68566/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5/volumes/kubernetes.io~projected/kube-api-access-h4z4z DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d/volumes/kubernetes.io~projected/kube-api-access-xkjjv DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf/volumes/kubernetes.io~projected/kube-api-access-dch9b DeviceMajor:0 DeviceMinor:358 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b00819ff-6b3d-4e36-995b-2d671e0dc9f6/volumes/kubernetes.io~projected/kube-api-access-5dlv2 DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a431ebd473676f2458785a54cd60c6116d7c401339a8f5af2cb6dbbdac4973e/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/edb1d4a1-0656-4417-8b7d-86a68db196ee/volumes/kubernetes.io~projected/kube-api-access-8mw4d DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aff3c5989d7f586de304608ba8eaced771e012ed3f4770980b7aeb7b8d3fa9ee/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:233 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61d8c282-03cc-4a20-a999-6cda0b14c87e/volumes/kubernetes.io~projected/kube-api-access-7j5qj DeviceMajor:0 DeviceMinor:345 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a22219af-1262-42a9-8cbe-8e6323e4375b/volumes/kubernetes.io~projected/kube-api-access-wd8sm DeviceMajor:0 DeviceMinor:384 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dad8ff83-33fe-4482-927e-8b70b1e72643/volumes/kubernetes.io~projected/kube-api-access-k45tm DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d762ba31bc07b465c2882275574c371615a22b61191d8c4c8ff5bc614c715e6d/userdata/shm DeviceMajor:0 DeviceMinor:99 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:03341514a396ff5 MacAddress:2a:69:f6:87:37:40 Speed:10000 Mtu:8901} {Name:03a2ab831fb7262 MacAddress:5a:98:8e:a9:e0:10 Speed:10000 Mtu:8901} {Name:040234a2b252b62 MacAddress:06:44:d8:22:66:54 Speed:10000 Mtu:8901} {Name:0483756c8af80f0 MacAddress:e2:62:3b:8c:d4:33 Speed:10000 Mtu:8901} {Name:0a431ebd473676f MacAddress:9a:60:b9:14:d6:51 Speed:10000 Mtu:8901} {Name:2cd11b588eab8bb MacAddress:ae:dd:ab:b2:7e:8c Speed:10000 Mtu:8901} {Name:71026b6e406cc80 MacAddress:32:40:0b:2c:7b:5f Speed:10000 Mtu:8901} {Name:bb50e1727539126 MacAddress:f2:c5:86:73:d3:4d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f9:8c:12:b7 Speed:0 Mtu:9001} {Name:br-int MacAddress:e6:3f:05:24:7b:5d Speed:0 Mtu:8901} {Name:d9a77ad916f5ee9 MacAddress:6a:f1:7e:40:2e:45 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f9:8c:12:b7 Speed:0 Mtu:9001} {Name:f0e2dd814d101ad MacAddress:46:8f:3c:1e:9e:a5 Speed:10000 Mtu:8901} {Name:f6282608a6482fa MacAddress:2a:74:b7:8f:a0:ed Speed:10000 Mtu:8901} {Name:fafcf58717120bc MacAddress:46:78:c9:a6:26:18 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:2e:af:e6:f0:c9:73 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:36:eb:b1:cd:d2:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.544960 6389 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.545006 6389 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.545349 6389 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.545371 6389 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-191.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.545511 6389 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.545523 6389 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.545542 6389 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.545571 6389 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.545720 6389 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.545961 6389 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.546094 6389 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.546130 6389 kubelet.go:493] "Attempting to sync node with API server" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.546148 6389 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.546166 6389 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.546184 6389 kubelet.go:405] "Adding apiserver pod source" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.546194 6389 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.547138 6389 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 14:12:04.548225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.547154 6389 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 14:12:04.550412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.550387 6389 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 14:12:04.554048 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554031 6389 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 14:12:04.554048 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554048 6389 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 14:12:04.554177 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554139 6389 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 14:12:04.554387 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554373 6389 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 14:12:04.554468 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554407 6389 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 14:12:04.554548 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554535 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 14:12:04.554598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554557 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 14:12:04.554598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554564 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 14:12:04.554598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554570 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 14:12:04.554598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554576 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 14:12:04.554598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554584 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 14:12:04.554598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554594 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 14:12:04.554815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554619 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 14:12:04.554815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554630 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 14:12:04.554815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554639 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 14:12:04.554815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554653 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 14:12:04.554815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554668 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 14:12:04.554815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554701 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 14:12:04.554815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.554709 6389 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 14:12:04.558849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.558831 6389 server.go:1263] "Started kubelet" Nov 03 14:12:04.559036 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.558950 6389 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 14:12:04.559123 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.559081 6389 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 14:12:04.559424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.559368 6389 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 14:12:04.559722 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.559705 6389 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 14:12:04.559879 ip-10-0-130-191 systemd[1]: Started Kubernetes Kubelet. Nov 03 14:12:04.560681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.560577 6389 server.go:317] "Adding debug handlers to kubelet server" Nov 03 14:12:04.565428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.564887 6389 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 14:12:04.565428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.564919 6389 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 14:12:04.565428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.565145 6389 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 14:05:53 +0000 UTC" deadline="2028-07-19 13:11:25.44737727 +0000 UTC" Nov 03 14:12:04.565428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.565196 6389 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14974h59m20.882186325s" Nov 03 14:12:04.565428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.565251 6389 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 14:12:04.565428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.565262 6389 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 14:12:04.565834 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:04.565415 6389 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-191.ec2.internal\" not found" Nov 03 14:12:04.566127 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.566106 6389 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 14:12:04.573871 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.573821 6389 factory.go:153] Registering CRI-O factory Nov 03 14:12:04.574002 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.573883 6389 factory.go:223] Registration of the crio container factory successfully Nov 03 14:12:04.574002 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.573955 6389 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 14:12:04.574002 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.573966 6389 factory.go:55] Registering systemd factory Nov 03 14:12:04.574002 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.573976 6389 factory.go:223] Registration of the systemd container factory successfully Nov 03 14:12:04.574002 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.573997 6389 factory.go:103] Registering Raw factory Nov 03 14:12:04.574229 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.574013 6389 manager.go:1199] Started watching for new ooms in manager Nov 03 14:12:04.574768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.574747 6389 manager.go:319] Starting recovery of all containers Nov 03 14:12:04.575961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.575822 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="977c941d-36f2-4c16-8396-95bdce627a74" volumeName="kubernetes.io/empty-dir/977c941d-36f2-4c16-8396-95bdce627a74-tmp-dir" seLinuxMountContext="" Nov 03 14:12:04.575961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.575866 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22219af-1262-42a9-8cbe-8e6323e4375b" volumeName="kubernetes.io/empty-dir/a22219af-1262-42a9-8cbe-8e6323e4375b-tmp" seLinuxMountContext="" Nov 03 14:12:04.575961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.575880 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f394a27c-5456-4b99-8b59-b89d25aae03b" volumeName="kubernetes.io/secret/f394a27c-5456-4b99-8b59-b89d25aae03b-cert" seLinuxMountContext="" Nov 03 14:12:04.575961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.575892 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f824baea-c6ee-409c-a1d3-e4f8b02bc6c5" volumeName="kubernetes.io/empty-dir/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-data-volume" seLinuxMountContext="" Nov 03 14:12:04.575961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.575906 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f824baea-c6ee-409c-a1d3-e4f8b02bc6c5" volumeName="kubernetes.io/secret/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 14:12:04.575961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.575919 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f394a27c-5456-4b99-8b59-b89d25aae03b" volumeName="kubernetes.io/projected/f394a27c-5456-4b99-8b59-b89d25aae03b-kube-api-access-xpp9t" seLinuxMountContext="" Nov 03 14:12:04.575961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.575931 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="608de1a1-c806-48bb-aecc-def971eca078" volumeName="kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 14:12:04.575961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.575944 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e380172-2ed6-4bd3-9fdb-83a694b400d0" volumeName="kubernetes.io/projected/8e380172-2ed6-4bd3-9fdb-83a694b400d0-kube-api-access-jhgqh" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.575969 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57fa4d4-1e9d-4a56-aba9-c6314ff12aea" volumeName="kubernetes.io/projected/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-kube-api-access-f2wbc" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.575981 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2a85509-0811-4aaf-a86d-b41e5ae5a9cc" volumeName="kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovnkube-config" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.575995 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="608de1a1-c806-48bb-aecc-def971eca078" volumeName="kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-secret-telemeter-client" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576011 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="977c941d-36f2-4c16-8396-95bdce627a74" volumeName="kubernetes.io/secret/977c941d-36f2-4c16-8396-95bdce627a74-metrics-tls" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576024 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22219af-1262-42a9-8cbe-8e6323e4375b" volumeName="kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-service-ca" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576036 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b00819ff-6b3d-4e36-995b-2d671e0dc9f6" volumeName="kubernetes.io/empty-dir/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-tuned" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576051 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6b9aee7-f7fb-4a15-8527-9ad5bda2523d" volumeName="kubernetes.io/projected/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-kube-api-access-xkjjv" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576063 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2a85509-0811-4aaf-a86d-b41e5ae5a9cc" volumeName="kubernetes.io/projected/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-kube-api-access-4k6zn" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576076 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="608de1a1-c806-48bb-aecc-def971eca078" volumeName="kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576095 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61d8c282-03cc-4a20-a999-6cda0b14c87e" volumeName="kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-service-ca" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576108 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61d8c282-03cc-4a20-a999-6cda0b14c87e" volumeName="kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-serving-cert" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576120 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61d8c282-03cc-4a20-a999-6cda0b14c87e" volumeName="kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-oauth-config" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576132 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="977c941d-36f2-4c16-8396-95bdce627a74" volumeName="kubernetes.io/configmap/977c941d-36f2-4c16-8396-95bdce627a74-config-volume" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576143 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="977c941d-36f2-4c16-8396-95bdce627a74" volumeName="kubernetes.io/projected/977c941d-36f2-4c16-8396-95bdce627a74-kube-api-access-g2q54" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576156 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22219af-1262-42a9-8cbe-8e6323e4375b" volumeName="kubernetes.io/projected/a22219af-1262-42a9-8cbe-8e6323e4375b-kube-api-access-wd8sm" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576170 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22219af-1262-42a9-8cbe-8e6323e4375b" volumeName="kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-oauth-config" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576200 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05a67278-34b4-462d-b4cb-9250800bf3db" volumeName="kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576214 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05a67278-34b4-462d-b4cb-9250800bf3db" volumeName="kubernetes.io/projected/05a67278-34b4-462d-b4cb-9250800bf3db-kube-api-access-wkjnc" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576227 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61d8c282-03cc-4a20-a999-6cda0b14c87e" volumeName="kubernetes.io/empty-dir/61d8c282-03cc-4a20-a999-6cda0b14c87e-tmp" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576249 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22219af-1262-42a9-8cbe-8e6323e4375b" volumeName="kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-serving-cert" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576275 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b586d272-c9c0-455f-aec9-333ae6d0c69a" volumeName="kubernetes.io/empty-dir/b586d272-c9c0-455f-aec9-333ae6d0c69a-tmp-dir" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576288 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f824baea-c6ee-409c-a1d3-e4f8b02bc6c5" volumeName="kubernetes.io/configmap/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 14:12:04.576367 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.576300 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57fa4d4-1e9d-4a56-aba9-c6314ff12aea" volumeName="kubernetes.io/configmap/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-metrics-client-ca" seLinuxMountContext="" Nov 03 14:12:04.578387 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578348 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05a67278-34b4-462d-b4cb-9250800bf3db" volumeName="kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 14:12:04.578487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578394 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="608de1a1-c806-48bb-aecc-def971eca078" volumeName="kubernetes.io/projected/608de1a1-c806-48bb-aecc-def971eca078-kube-api-access-tjxcm" seLinuxMountContext="" Nov 03 14:12:04.578487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578408 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="608de1a1-c806-48bb-aecc-def971eca078" volumeName="kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 14:12:04.578487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578423 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e380172-2ed6-4bd3-9fdb-83a694b400d0" volumeName="kubernetes.io/secret/8e380172-2ed6-4bd3-9fdb-83a694b400d0-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 14:12:04.578487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578439 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af2b7bc6-b143-4ffd-b58c-3ca6f13eec56" volumeName="kubernetes.io/empty-dir/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-volume-directive-shadow" seLinuxMountContext="" Nov 03 14:12:04.578487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578451 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af2b7bc6-b143-4ffd-b58c-3ca6f13eec56" volumeName="kubernetes.io/secret/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 14:12:04.578487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578464 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="608de1a1-c806-48bb-aecc-def971eca078" volumeName="kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-telemeter-client-tls" seLinuxMountContext="" Nov 03 14:12:04.578487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578478 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22219af-1262-42a9-8cbe-8e6323e4375b" volumeName="kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-console-config" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578490 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af2b7bc6-b143-4ffd-b58c-3ca6f13eec56" volumeName="kubernetes.io/secret/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578504 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b00819ff-6b3d-4e36-995b-2d671e0dc9f6" volumeName="kubernetes.io/empty-dir/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-tmp" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578519 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b00819ff-6b3d-4e36-995b-2d671e0dc9f6" volumeName="kubernetes.io/projected/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-kube-api-access-5dlv2" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578534 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d153593a-c25f-4bef-acca-ea7e50b32c87" volumeName="kubernetes.io/configmap/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-daemon-config" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578548 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dad8ff83-33fe-4482-927e-8b70b1e72643" volumeName="kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578561 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2a85509-0811-4aaf-a86d-b41e5ae5a9cc" volumeName="kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-env-overrides" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578574 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e380172-2ed6-4bd3-9fdb-83a694b400d0" volumeName="kubernetes.io/configmap/8e380172-2ed6-4bd3-9fdb-83a694b400d0-metrics-client-ca" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578587 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0694b8ee-14f6-495d-a662-8f06bdfe1c66" volumeName="kubernetes.io/projected/0694b8ee-14f6-495d-a662-8f06bdfe1c66-kube-api-access-m7d5k" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578602 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61d8c282-03cc-4a20-a999-6cda0b14c87e" volumeName="kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-oauth-serving-cert" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578639 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="905b521a-2b6a-4fb6-ba5b-33311a2613b0" volumeName="kubernetes.io/configmap/905b521a-2b6a-4fb6-ba5b-33311a2613b0-konnectivity-ca" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578651 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22219af-1262-42a9-8cbe-8e6323e4375b" volumeName="kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-trusted-ca-bundle" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578664 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="608de1a1-c806-48bb-aecc-def971eca078" volumeName="kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-federate-client-tls" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578677 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="905b521a-2b6a-4fb6-ba5b-33311a2613b0" volumeName="kubernetes.io/secret/905b521a-2b6a-4fb6-ba5b-33311a2613b0-agent-certs" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578691 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af2b7bc6-b143-4ffd-b58c-3ca6f13eec56" volumeName="kubernetes.io/configmap/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-metrics-client-ca" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578719 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edb1d4a1-0656-4417-8b7d-86a68db196ee" volumeName="kubernetes.io/configmap/edb1d4a1-0656-4417-8b7d-86a68db196ee-serviceca" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578731 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc214996-6ee2-4bf3-a2cd-49cd4eceaa09" volumeName="kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578742 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05a67278-34b4-462d-b4cb-9250800bf3db" volumeName="kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-cni-binary-copy" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578763 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57fa4d4-1e9d-4a56-aba9-c6314ff12aea" volumeName="kubernetes.io/secret/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-tls" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578776 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d153593a-c25f-4bef-acca-ea7e50b32c87" volumeName="kubernetes.io/configmap/d153593a-c25f-4bef-acca-ea7e50b32c87-cni-binary-copy" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578787 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dad8ff83-33fe-4482-927e-8b70b1e72643" volumeName="kubernetes.io/projected/dad8ff83-33fe-4482-927e-8b70b1e72643-kube-api-access-k45tm" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578800 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f824baea-c6ee-409c-a1d3-e4f8b02bc6c5" volumeName="kubernetes.io/projected/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-kube-api-access-h4z4z" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578811 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="608de1a1-c806-48bb-aecc-def971eca078" volumeName="kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-metrics-client-ca" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578822 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22219af-1262-42a9-8cbe-8e6323e4375b" volumeName="kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-oauth-serving-cert" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578836 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57fa4d4-1e9d-4a56-aba9-c6314ff12aea" volumeName="kubernetes.io/configmap/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578847 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57fa4d4-1e9d-4a56-aba9-c6314ff12aea" volumeName="kubernetes.io/secret/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578858 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2a85509-0811-4aaf-a86d-b41e5ae5a9cc" volumeName="kubernetes.io/secret/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578871 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40e20104-e97c-41b2-8530-abb2dec148a2" volumeName="kubernetes.io/projected/40e20104-e97c-41b2-8530-abb2dec148a2-kube-api-access-kq4pb" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578882 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af2b7bc6-b143-4ffd-b58c-3ca6f13eec56" volumeName="kubernetes.io/projected/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-api-access-47v7j" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578893 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d153593a-c25f-4bef-acca-ea7e50b32c87" volumeName="kubernetes.io/projected/d153593a-c25f-4bef-acca-ea7e50b32c87-kube-api-access-z2mvt" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578906 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6b9aee7-f7fb-4a15-8527-9ad5bda2523d" volumeName="kubernetes.io/configmap/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-iptables-alerter-script" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578918 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edb1d4a1-0656-4417-8b7d-86a68db196ee" volumeName="kubernetes.io/projected/edb1d4a1-0656-4417-8b7d-86a68db196ee-kube-api-access-8mw4d" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578929 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57fa4d4-1e9d-4a56-aba9-c6314ff12aea" volumeName="kubernetes.io/empty-dir/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-textfile" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578943 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0694b8ee-14f6-495d-a662-8f06bdfe1c66" volumeName="kubernetes.io/empty-dir/0694b8ee-14f6-495d-a662-8f06bdfe1c66-tmp" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578955 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61d8c282-03cc-4a20-a999-6cda0b14c87e" volumeName="kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-config" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578967 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f1c105f-a7fa-4aeb-b8de-a413f5def9bf" volumeName="kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578978 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e380172-2ed6-4bd3-9fdb-83a694b400d0" volumeName="kubernetes.io/secret/8e380172-2ed6-4bd3-9fdb-83a694b400d0-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.578989 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af2b7bc6-b143-4ffd-b58c-3ca6f13eec56" volumeName="kubernetes.io/configmap/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.579000 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b586d272-c9c0-455f-aec9-333ae6d0c69a" volumeName="kubernetes.io/projected/b586d272-c9c0-455f-aec9-333ae6d0c69a-kube-api-access-7d4lt" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.579014 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2a85509-0811-4aaf-a86d-b41e5ae5a9cc" volumeName="kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovnkube-script-lib" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.579025 6389 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61d8c282-03cc-4a20-a999-6cda0b14c87e" volumeName="kubernetes.io/projected/61d8c282-03cc-4a20-a999-6cda0b14c87e-kube-api-access-7j5qj" seLinuxMountContext="" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.579037 6389 reconstruct.go:97] "Volume reconstruction finished" Nov 03 14:12:04.580640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.579044 6389 reconciler.go:29] "Reconciler: start to sync state" Nov 03 14:12:04.608598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.608522 6389 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 14:12:04.610257 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.610232 6389 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 14:12:04.610257 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.610257 6389 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 14:12:04.610421 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.610279 6389 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 14:12:04.610421 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:04.610320 6389 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 14:12:04.610527 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.610516 6389 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 14:12:04.610718 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.610698 6389 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:12:04.622002 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.621978 6389 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:12:04.634694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.634663 6389 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:12:04.634874 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.634857 6389 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:12:04.657937 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.657896 6389 generic.go:356] "Generic (PLEG): container finished" podID="b57fa4d4-1e9d-4a56-aba9-c6314ff12aea" containerID="3e0b84076fdb124176e5b50f121f772b65c38be0fbf4f778a4a8dcf2bc6c1c3b" exitCode=0 Nov 03 14:12:04.672656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.672594 6389 generic.go:356] "Generic (PLEG): container finished" podID="3c5016cec399e6887c94ab5ea8dbdf29" containerID="1a8e0dbc8ce266e152d48a70ee62821ad8999e34386a76bbc3d292a321dad5ef" exitCode=0 Nov 03 14:12:04.694231 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.694176 6389 generic.go:356] "Generic (PLEG): container finished" podID="05a67278-34b4-462d-b4cb-9250800bf3db" containerID="fc3d9b67547ebc0e6ae822c6893bcd31c980e9a208f50a154274cdaed19a6302" exitCode=0 Nov 03 14:12:04.694231 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.694201 6389 generic.go:356] "Generic (PLEG): container finished" podID="05a67278-34b4-462d-b4cb-9250800bf3db" containerID="c448caa6c81bc55ba0a5b992c35a7721ee924452cc0821303b60830e923b5394" exitCode=0 Nov 03 14:12:04.694231 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.694211 6389 generic.go:356] "Generic (PLEG): container finished" podID="05a67278-34b4-462d-b4cb-9250800bf3db" containerID="1b8c238ee2d4663391c4e99baeba42226991566d44c8896deb8073ed68c10783" exitCode=0 Nov 03 14:12:04.694231 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.694221 6389 generic.go:356] "Generic (PLEG): container finished" podID="05a67278-34b4-462d-b4cb-9250800bf3db" containerID="963aabbb5fe1c30f57377f07047a35d49dad86b249a6046ffe2bf2c7a97e7397" exitCode=0 Nov 03 14:12:04.694231 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.694229 6389 generic.go:356] "Generic (PLEG): container finished" podID="05a67278-34b4-462d-b4cb-9250800bf3db" containerID="e9cde02f06b5a90d447e5c502d0fbc4efa76d72f604f4bb7c78337db3f12fcb4" exitCode=0 Nov 03 14:12:04.694231 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.694238 6389 generic.go:356] "Generic (PLEG): container finished" podID="05a67278-34b4-462d-b4cb-9250800bf3db" containerID="7cce28818517e048d93a205f99151f528c121695e1ccd1bdbeade5577d9a47d6" exitCode=0 Nov 03 14:12:04.708414 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.708396 6389 manager.go:324] Recovery completed Nov 03 14:12:04.710534 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:04.710514 6389 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 14:12:04.740395 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740367 6389 cpu_manager.go:226] "Starting" policy="none" Nov 03 14:12:04.740395 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740389 6389 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 14:12:04.740596 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740413 6389 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 14:12:04.740596 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740566 6389 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 14:12:04.740596 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740578 6389 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 14:12:04.740762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740601 6389 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 14:12:04.740762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740624 6389 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 14:12:04.740762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740631 6389 policy_none.go:50] "Start" Nov 03 14:12:04.740762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740650 6389 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 14:12:04.740762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740664 6389 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 14:12:04.740953 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740842 6389 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 14:12:04.740953 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740853 6389 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 14:12:04.740953 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.740861 6389 policy_none.go:44] "Start" Nov 03 14:12:04.746499 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.746467 6389 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 14:12:04.749831 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.749813 6389 manager.go:341] "Starting Device Plugin manager" Nov 03 14:12:04.750558 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:04.749845 6389 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 14:12:04.750666 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.750564 6389 server.go:87] "Starting device plugin registration server" Nov 03 14:12:04.750946 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.750927 6389 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 14:12:04.751043 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.750945 6389 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 14:12:04.752099 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.752080 6389 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 14:12:04.752230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.752182 6389 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 14:12:04.752230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.752216 6389 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 14:12:04.752230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.752223 6389 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 14:12:04.752567 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.752386 6389 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T14:12:04.7522053Z","UUID":"a7e19c79-3441-4b8f-9197-3145ed916c19","Handler":null,"Name":"","Endpoint":""} Nov 03 14:12:04.753059 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:04.753038 6389 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 14:12:04.754280 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.754192 6389 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 14:12:04.754280 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.754220 6389 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 14:12:04.851083 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.851054 6389 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:12:04.853551 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.853527 6389 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:12:04.853700 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.853568 6389 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:12:04.853700 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.853582 6389 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-191.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:12:04.853700 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.853660 6389 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-191.ec2.internal" Nov 03 14:12:04.893727 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.893657 6389 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-191.ec2.internal" Nov 03 14:12:04.893889 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.893737 6389 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-191.ec2.internal" Nov 03 14:12:04.911619 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.911559 6389 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal"] Nov 03 14:12:04.912365 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.912318 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" event={"ID":"3c5016cec399e6887c94ab5ea8dbdf29","Type":"ContainerStarted","Data":"5aaacc2082e87350c4c9aad261d28a42e57c300877933879c2dfe03e0d9800d7"} Nov 03 14:12:04.912365 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.912358 6389 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:12:04.912568 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.912376 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" event={"ID":"3c5016cec399e6887c94ab5ea8dbdf29","Type":"ContainerDied","Data":"1a8e0dbc8ce266e152d48a70ee62821ad8999e34386a76bbc3d292a321dad5ef"} Nov 03 14:12:04.912568 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.912399 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" event={"ID":"3c5016cec399e6887c94ab5ea8dbdf29","Type":"ContainerStarted","Data":"69fc2d0a5d65d549a0169817881ddf3a96f96b3096dadd736528ef36dfbfe900"} Nov 03 14:12:04.912568 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.912469 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" event={"ID":"b01771a71144616672fedcb36938a552","Type":"ContainerStarted","Data":"7ac05325799f5f60816b433404b7b4118942f1daecdadb85c75eb7196e30bbfb"} Nov 03 14:12:04.912568 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.912490 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" event={"ID":"b01771a71144616672fedcb36938a552","Type":"ContainerStarted","Data":"999b1867c7515b06d1e6db3f17fe4b82cd262b99539615faabe4170a1711fd9b"} Nov 03 14:12:04.912759 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.912598 6389 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:12:04.936104 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.936080 6389 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 14:12:04.936241 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:04.936123 6389 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-191.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:12:04.937539 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:04.937510 6389 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 14:12:04.937674 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:04.937554 6389 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.011915 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.011876 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b01771a71144616672fedcb36938a552-config\") pod \"kube-apiserver-proxy-ip-10-0-130-191.ec2.internal\" (UID: \"b01771a71144616672fedcb36938a552\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.011915 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.011921 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c5016cec399e6887c94ab5ea8dbdf29-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" (UID: \"3c5016cec399e6887c94ab5ea8dbdf29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.012136 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.011948 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c5016cec399e6887c94ab5ea8dbdf29-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" (UID: \"3c5016cec399e6887c94ab5ea8dbdf29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.112480 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.112444 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c5016cec399e6887c94ab5ea8dbdf29-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" (UID: \"3c5016cec399e6887c94ab5ea8dbdf29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.112720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.112486 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c5016cec399e6887c94ab5ea8dbdf29-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" (UID: \"3c5016cec399e6887c94ab5ea8dbdf29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.112720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.112565 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c5016cec399e6887c94ab5ea8dbdf29-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" (UID: \"3c5016cec399e6887c94ab5ea8dbdf29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.112720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.112602 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b01771a71144616672fedcb36938a552-config\") pod \"kube-apiserver-proxy-ip-10-0-130-191.ec2.internal\" (UID: \"b01771a71144616672fedcb36938a552\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.112720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.112566 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c5016cec399e6887c94ab5ea8dbdf29-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" (UID: \"3c5016cec399e6887c94ab5ea8dbdf29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.112720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.112669 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b01771a71144616672fedcb36938a552-config\") pod \"kube-apiserver-proxy-ip-10-0-130-191.ec2.internal\" (UID: \"b01771a71144616672fedcb36938a552\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.546347 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.546312 6389 apiserver.go:52] "Watching apiserver" Nov 03 14:12:05.596628 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.596574 6389 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 14:12:05.598306 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.598258 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zbs97","kube-system/global-pull-secret-syncer-tjsp9","openshift-dns/node-resolver-qnhkx","kube-system/konnectivity-agent-frkjs","openshift-cluster-node-tuning-operator/tuned-747sw","openshift-console/console-84c65c5d7-j9sv7","openshift-insights/insights-runtime-extractor-wqbbs","openshift-network-diagnostics/network-check-target-phmgc","openshift-console/console-7c959ffc98-lphmd","openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal","openshift-multus/multus-additional-cni-plugins-99kd6","openshift-network-operator/iptables-alerter-9pdj9","openshift-image-registry/node-ca-dlwbh","openshift-multus/multus-vlblr","openshift-ingress-canary/ingress-canary-2twr6","openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs","openshift-dns/dns-default-j6lgv","openshift-monitoring/telemeter-client-65979fb8bc-sh6c8","openshift-multus/network-metrics-daemon-r5lc9","openshift-ovn-kubernetes/ovnkube-node-7j9ks","kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal","openshift-console/downloads-6dd789c8b7-dzpfq"] Nov 03 14:12:05.619892 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.619865 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 14:12:05.620026 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.619946 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-95wl7\"" Nov 03 14:12:05.620026 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.619865 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2ft6v\"" Nov 03 14:12:05.620026 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620015 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xm8pf\"" Nov 03 14:12:05.620216 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620198 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 14:12:05.620279 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620267 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 14:12:05.620328 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620274 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qhrll\"" Nov 03 14:12:05.620328 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620322 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 14:12:05.620403 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620384 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 14:12:05.620403 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620396 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lqmd6\"" Nov 03 14:12:05.620490 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620441 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 14:12:05.620490 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620446 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:12:05.620590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620492 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 14:12:05.620590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620532 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 14:12:05.620590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620555 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 14:12:05.620590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620572 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 14:12:05.620794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620598 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 14:12:05.620794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620628 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 14:12:05.620794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620601 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 14:12:05.620794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620704 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 14:12:05.620794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620733 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 14:12:05.620794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620762 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 14:12:05.620794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620772 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:12:05.621062 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620825 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trbj6\"" Nov 03 14:12:05.621062 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620840 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 14:12:05.621062 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620861 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 14:12:05.621062 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.620914 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 14:12:05.621957 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.621934 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rmj5t\"" Nov 03 14:12:05.622072 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.621996 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qgxkz\"" Nov 03 14:12:05.622250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.622185 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ml487\"" Nov 03 14:12:05.622571 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.622278 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 14:12:05.622571 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.622399 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 14:12:05.622571 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.622446 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 14:12:05.622571 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.622455 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hlrhs\"" Nov 03 14:12:05.622571 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.622400 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 14:12:05.622571 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.622482 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r6xhc\"" Nov 03 14:12:05.622912 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.622699 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tb44c\"" Nov 03 14:12:05.622968 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.622947 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 14:12:05.623060 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623038 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 14:12:05.623119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623039 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 14:12:05.623171 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623142 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 14:12:05.623171 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623157 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 14:12:05.623263 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623175 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 14:12:05.623263 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623235 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 14:12:05.623378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623361 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 14:12:05.623378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623373 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 14:12:05.623485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623388 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q495v\"" Nov 03 14:12:05.623485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623417 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 14:12:05.623485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623444 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 14:12:05.623485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623471 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bgwlk\"" Nov 03 14:12:05.623653 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.623506 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 14:12:05.624401 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.624378 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 14:12:05.629292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.629267 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 14:12:05.629395 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.629374 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 14:12:05.629459 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.629397 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 14:12:05.629596 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.629571 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 14:12:05.629719 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.629602 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 14:12:05.629783 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.629737 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 14:12:05.629841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.629784 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 14:12:05.629841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.629837 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 14:12:05.629935 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.629845 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 14:12:05.629935 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.629906 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 14:12:05.630030 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.629997 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 14:12:05.631396 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.631376 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 14:12:05.631396 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.631379 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 14:12:05.631824 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.631807 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 14:12:05.631824 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.631815 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 14:12:05.631967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.631818 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 14:12:05.632422 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632398 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 14:12:05.632422 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632406 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghsgd\"" Nov 03 14:12:05.632570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632461 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 14:12:05.632570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632463 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bvxcr\"" Nov 03 14:12:05.632570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632483 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 14:12:05.632570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632512 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 14:12:05.632570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632554 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 14:12:05.632841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632641 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 14:12:05.632841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632641 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rp88m\"" Nov 03 14:12:05.632841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632671 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 14:12:05.632841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632706 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-cv9df\"" Nov 03 14:12:05.632841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632735 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fn7sx\"" Nov 03 14:12:05.632841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632778 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kfj2l\"" Nov 03 14:12:05.633069 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632863 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zjlcb\"" Nov 03 14:12:05.633069 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632879 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 14:12:05.633069 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.632967 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 14:12:05.638997 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.638979 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 14:12:05.639115 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.639097 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 14:12:05.668914 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.668891 6389 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 14:12:05.715950 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.715923 6389 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.716094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.715974 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:12:05.716094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716019 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-oauth-config\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.716094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716054 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edb1d4a1-0656-4417-8b7d-86a68db196ee-host\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:12:05.716094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716070 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j5qj\" (UniqueName: \"kubernetes.io/projected/61d8c282-03cc-4a20-a999-6cda0b14c87e-kube-api-access-7j5qj\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.716094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716077 6389 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.716094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716095 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-sys\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.716389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716129 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-iptables-alerter-script\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:12:05.716389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716153 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-host-slash\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:12:05.716389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716180 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-kubelet-config\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:12:05.716389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716209 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-metrics-client-ca\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.716389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716238 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e380172-2ed6-4bd3-9fdb-83a694b400d0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:12:05.716389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716272 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.716389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716304 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-oauth-config\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.716389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716304 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad8ff83-33fe-4482-927e-8b70b1e72643-metrics-certs\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:12:05.716754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716392 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-crio-socket\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:12:05.716754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716514 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:12:05.716754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716636 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-metrics-client-ca\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.716754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716640 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-iptables-alerter-script\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:12:05.716754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716696 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-socket-dir-parent\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.716754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716723 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-kubelet\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.716754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716724 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e380172-2ed6-4bd3-9fdb-83a694b400d0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:12:05.717077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716759 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-cnibin\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.717077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716778 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:12:05.717077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716808 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.717077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716833 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-serving-cert\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.717077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.716987 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.717077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717019 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-service-ca\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.717077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717026 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-serving-cert\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.717378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717139 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-socket-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.717378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717179 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-telemeter-client-tls\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.717378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717185 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-service-ca\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.717378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717212 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxcm\" (UniqueName: \"kubernetes.io/projected/608de1a1-c806-48bb-aecc-def971eca078-kube-api-access-tjxcm\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.717378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717239 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-hostroot\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.717594 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717451 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-service-ca\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.717594 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717483 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/905b521a-2b6a-4fb6-ba5b-33311a2613b0-agent-certs\") pod \"konnectivity-agent-frkjs\" (UID: \"905b521a-2b6a-4fb6-ba5b-33311a2613b0\") " pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:12:05.717594 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717506 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-registration-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.717594 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717519 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-telemeter-client-tls\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.717594 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717534 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.717594 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717560 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-root\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.717594 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717584 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-multus-certs\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.717903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717623 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-device-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.717903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717650 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/905b521a-2b6a-4fb6-ba5b-33311a2613b0-konnectivity-ca\") pod \"konnectivity-agent-frkjs\" (UID: \"905b521a-2b6a-4fb6-ba5b-33311a2613b0\") " pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:12:05.717903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717678 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovnkube-config\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.717903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717705 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-service-ca\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.717903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717711 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:12:05.717903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717745 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-dbus\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:12:05.717903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717773 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-secret-telemeter-client\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.717903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717788 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/905b521a-2b6a-4fb6-ba5b-33311a2613b0-agent-certs\") pod \"konnectivity-agent-frkjs\" (UID: \"905b521a-2b6a-4fb6-ba5b-33311a2613b0\") " pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:12:05.717903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717798 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-kubelet\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.717903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717825 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.717903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717851 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-lib-modules\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.717903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.717895 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:12:05.718408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718062 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovnkube-config\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.718408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718087 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-secret-telemeter-client\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.718408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718123 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-sys-fs\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.718408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718185 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.718408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718231 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/905b521a-2b6a-4fb6-ba5b-33311a2613b0-konnectivity-ca\") pod \"konnectivity-agent-frkjs\" (UID: \"905b521a-2b6a-4fb6-ba5b-33311a2613b0\") " pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:12:05.718408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718246 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k6zn\" (UniqueName: \"kubernetes.io/projected/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-kube-api-access-4k6zn\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.718408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718285 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-data-volume\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:12:05.718408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718307 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.718768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718449 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:12:05.718768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718520 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-data-volume\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:12:05.718768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718581 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47v7j\" (UniqueName: \"kubernetes.io/projected/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-api-access-47v7j\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:12:05.718768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718668 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-netns\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.718768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718757 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:12:05.718984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718762 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovn-node-metrics-cert\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.718984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.718880 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:12:05.719078 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719012 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovn-node-metrics-cert\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.719078 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719028 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-textfile\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.719149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719069 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:12:05.719149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719114 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-textfile\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.719149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719144 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-etc-kubernetes\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.719256 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719204 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2mvt\" (UniqueName: \"kubernetes.io/projected/d153593a-c25f-4bef-acca-ea7e50b32c87-kube-api-access-z2mvt\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.719256 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719242 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-systemd-units\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.719375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719356 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-run-netns\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.719500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719441 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-var-lib-kubelet\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.719500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719465 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-serving-certs-ca-bundle\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.719500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719481 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-cnibin\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.719660 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719499 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.719660 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719559 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-serving-cert\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.719660 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719581 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e380172-2ed6-4bd3-9fdb-83a694b400d0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:12:05.719793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719679 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e380172-2ed6-4bd3-9fdb-83a694b400d0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:12:05.719793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719697 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608de1a1-c806-48bb-aecc-def971eca078-serving-certs-ca-bundle\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.719793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719711 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-kubernetes\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.719793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719739 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c941d-36f2-4c16-8396-95bdce627a74-tmp-dir\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:12:05.719793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719763 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2q54\" (UniqueName: \"kubernetes.io/projected/977c941d-36f2-4c16-8396-95bdce627a74-kube-api-access-g2q54\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:12:05.720044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719795 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhgqh\" (UniqueName: \"kubernetes.io/projected/8e380172-2ed6-4bd3-9fdb-83a694b400d0-kube-api-access-jhgqh\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:12:05.720044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719828 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e380172-2ed6-4bd3-9fdb-83a694b400d0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:12:05.720044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719838 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-log-socket\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.720044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719864 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-serving-cert\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.720044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719867 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-cni-netd\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.720044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719907 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c941d-36f2-4c16-8396-95bdce627a74-tmp-dir\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:12:05.720044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719922 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e380172-2ed6-4bd3-9fdb-83a694b400d0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:12:05.720044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719938 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-cni-bin\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.720044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.719984 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkjnc\" (UniqueName: \"kubernetes.io/projected/05a67278-34b4-462d-b4cb-9250800bf3db-kube-api-access-wkjnc\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.720044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720016 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a22219af-1262-42a9-8cbe-8e6323e4375b-tmp\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.720044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720036 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-accelerators-collector-config\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.720044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720054 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-cni-multus\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720076 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-oauth-config\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720100 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq4pb\" (UniqueName: \"kubernetes.io/projected/40e20104-e97c-41b2-8530-abb2dec148a2-kube-api-access-kq4pb\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720124 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720151 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720180 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/977c941d-36f2-4c16-8396-95bdce627a74-config-volume\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720203 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-modprobe-d\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720208 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a22219af-1262-42a9-8cbe-8e6323e4375b-tmp\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720229 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysctl-conf\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720274 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7d5k\" (UniqueName: \"kubernetes.io/projected/0694b8ee-14f6-495d-a662-8f06bdfe1c66-kube-api-access-m7d5k\") pod \"downloads-6dd789c8b7-dzpfq\" (UID: \"0694b8ee-14f6-495d-a662-8f06bdfe1c66\") " pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720406 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-cni-bin\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720460 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovnkube-script-lib\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720507 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-accelerators-collector-config\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720582 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-oauth-config\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.720593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720591 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/977c941d-36f2-4c16-8396-95bdce627a74-config-volume\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720650 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-tuned\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720723 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720742 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlv2\" (UniqueName: \"kubernetes.io/projected/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-kube-api-access-5dlv2\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720754 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-tuned\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720793 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wbc\" (UniqueName: \"kubernetes.io/projected/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-kube-api-access-f2wbc\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720819 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-systemd\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720835 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-ovnkube-script-lib\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720879 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-tmp\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720918 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-system-cni-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720947 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-system-cni-dir\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.720975 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721002 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-federate-client-tls\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721006 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-tmp\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721037 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-console-config\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721068 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8sm\" (UniqueName: \"kubernetes.io/projected/a22219af-1262-42a9-8cbe-8e6323e4375b-kube-api-access-wd8sm\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721232 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-federate-client-tls\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721245 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-console-config\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.721291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721278 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dch9b\" (UniqueName: \"kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b\") pod \"network-check-target-phmgc\" (UID: \"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf\") " pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721353 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-original-pull-secret\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721401 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-tls\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721442 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721481 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-sys\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721507 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-slash\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721539 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysctl-d\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721564 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-systemd\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721629 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-cni-binary-copy\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721636 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-tls\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721659 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-env-overrides\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721684 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-wtmp\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721727 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-cni-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721749 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b586d272-c9c0-455f-aec9-333ae6d0c69a-tmp-dir\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721787 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-oauth-serving-cert\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721815 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-env-overrides\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721820 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mw4d\" (UniqueName: \"kubernetes.io/projected/edb1d4a1-0656-4417-8b7d-86a68db196ee-kube-api-access-8mw4d\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721861 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b586d272-c9c0-455f-aec9-333ae6d0c69a-tmp-dir\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721946 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b586d272-c9c0-455f-aec9-333ae6d0c69a-hosts-file\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.721980 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpp9t\" (UniqueName: \"kubernetes.io/projected/f394a27c-5456-4b99-8b59-b89d25aae03b-kube-api-access-xpp9t\") pod \"ingress-canary-2twr6\" (UID: \"f394a27c-5456-4b99-8b59-b89d25aae03b\") " pod="openshift-ingress-canary/ingress-canary-2twr6" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722007 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-run\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722149 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-cni-binary-copy\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722155 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-oauth-serving-cert\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722167 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-conf-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722243 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61d8c282-03cc-4a20-a999-6cda0b14c87e-tmp\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722287 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722349 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61d8c282-03cc-4a20-a999-6cda0b14c87e-tmp\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.722406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722403 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4z4z\" (UniqueName: \"kubernetes.io/projected/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-kube-api-access-h4z4z\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722553 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k45tm\" (UniqueName: \"kubernetes.io/projected/dad8ff83-33fe-4482-927e-8b70b1e72643-kube-api-access-k45tm\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722579 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/608de1a1-c806-48bb-aecc-def971eca078-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722651 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-daemon-config\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722766 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c941d-36f2-4c16-8396-95bdce627a74-metrics-tls\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722791 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edb1d4a1-0656-4417-8b7d-86a68db196ee-serviceca\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722838 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0694b8ee-14f6-495d-a662-8f06bdfe1c66-tmp\") pod \"downloads-6dd789c8b7-dzpfq\" (UID: \"0694b8ee-14f6-495d-a662-8f06bdfe1c66\") " pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722870 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f394a27c-5456-4b99-8b59-b89d25aae03b-cert\") pod \"ingress-canary-2twr6\" (UID: \"f394a27c-5456-4b99-8b59-b89d25aae03b\") " pod="openshift-ingress-canary/ingress-canary-2twr6" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.722896 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-config\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723059 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/977c941d-36f2-4c16-8396-95bdce627a74-metrics-tls\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723105 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-config\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723106 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723173 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0694b8ee-14f6-495d-a662-8f06bdfe1c66-tmp\") pod \"downloads-6dd789c8b7-dzpfq\" (UID: \"0694b8ee-14f6-495d-a662-8f06bdfe1c66\") " pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723217 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f394a27c-5456-4b99-8b59-b89d25aae03b-cert\") pod \"ingress-canary-2twr6\" (UID: \"f394a27c-5456-4b99-8b59-b89d25aae03b\") " pod="openshift-ingress-canary/ingress-canary-2twr6" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723224 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d4lt\" (UniqueName: \"kubernetes.io/projected/b586d272-c9c0-455f-aec9-333ae6d0c69a-kube-api-access-7d4lt\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723265 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-oauth-serving-cert\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723392 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-metrics-client-ca\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723481 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-daemon-config\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723485 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-k8s-cni-cncf-io\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723528 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-oauth-serving-cert\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723646 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-metrics-client-ca\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723682 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-node-log\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.723717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723701 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723764 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723802 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysconfig\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723831 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edb1d4a1-0656-4417-8b7d-86a68db196ee-serviceca\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723840 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723894 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-etc-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723962 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.723995 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-host\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.724024 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d153593a-c25f-4bef-acca-ea7e50b32c87-cni-binary-copy\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.724064 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-trusted-ca-bundle\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.724085 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05a67278-34b4-462d-b4cb-9250800bf3db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.724129 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjjv\" (UniqueName: \"kubernetes.io/projected/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-kube-api-access-xkjjv\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.724492 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-var-lib-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.724749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.724703 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-ovn\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.725412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.724772 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:12:05.725412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.725058 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-trusted-ca-bundle\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.725412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.725138 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:12:05.725412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.725189 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d153593a-c25f-4bef-acca-ea7e50b32c87-cni-binary-copy\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.737536 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.737515 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j5qj\" (UniqueName: \"kubernetes.io/projected/61d8c282-03cc-4a20-a999-6cda0b14c87e-kube-api-access-7j5qj\") pod \"console-84c65c5d7-j9sv7\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:05.739672 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.739647 6389 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 14:12:05.739755 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:05.739695 6389 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-191.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.741627 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.741582 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxcm\" (UniqueName: \"kubernetes.io/projected/608de1a1-c806-48bb-aecc-def971eca078-kube-api-access-tjxcm\") pod \"telemeter-client-65979fb8bc-sh6c8\" (UID: \"608de1a1-c806-48bb-aecc-def971eca078\") " pod="openshift-monitoring/telemeter-client-65979fb8bc-sh6c8" Nov 03 14:12:05.741738 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.741689 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k6zn\" (UniqueName: \"kubernetes.io/projected/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-kube-api-access-4k6zn\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.743275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.743255 6389 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 14:12:05.743362 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:05.743301 6389 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-191.ec2.internal" Nov 03 14:12:05.744756 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.744729 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2mvt\" (UniqueName: \"kubernetes.io/projected/d153593a-c25f-4bef-acca-ea7e50b32c87-kube-api-access-z2mvt\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.745549 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.745530 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpp9t\" (UniqueName: \"kubernetes.io/projected/f394a27c-5456-4b99-8b59-b89d25aae03b-kube-api-access-xpp9t\") pod \"ingress-canary-2twr6\" (UID: \"f394a27c-5456-4b99-8b59-b89d25aae03b\") " pod="openshift-ingress-canary/ingress-canary-2twr6" Nov 03 14:12:05.746686 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.746665 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k45tm\" (UniqueName: \"kubernetes.io/projected/dad8ff83-33fe-4482-927e-8b70b1e72643-kube-api-access-k45tm\") pod \"network-metrics-daemon-r5lc9\" (UID: \"dad8ff83-33fe-4482-927e-8b70b1e72643\") " pod="openshift-multus/network-metrics-daemon-r5lc9" Nov 03 14:12:05.747304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.747284 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8sm\" (UniqueName: \"kubernetes.io/projected/a22219af-1262-42a9-8cbe-8e6323e4375b-kube-api-access-wd8sm\") pod \"console-7c959ffc98-lphmd\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:05.748225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.748194 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjjv\" (UniqueName: \"kubernetes.io/projected/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-kube-api-access-xkjjv\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:12:05.748361 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.748338 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mw4d\" (UniqueName: \"kubernetes.io/projected/edb1d4a1-0656-4417-8b7d-86a68db196ee-kube-api-access-8mw4d\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:12:05.748563 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.748485 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47v7j\" (UniqueName: \"kubernetes.io/projected/af2b7bc6-b143-4ffd-b58c-3ca6f13eec56-kube-api-access-47v7j\") pod \"kube-state-metrics-5dc8d7d89b-6k8f7\" (UID: \"af2b7bc6-b143-4ffd-b58c-3ca6f13eec56\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6k8f7" Nov 03 14:12:05.748563 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.748551 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhgqh\" (UniqueName: \"kubernetes.io/projected/8e380172-2ed6-4bd3-9fdb-83a694b400d0-kube-api-access-jhgqh\") pod \"openshift-state-metrics-7bd7489957-jsdph\" (UID: \"8e380172-2ed6-4bd3-9fdb-83a694b400d0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jsdph" Nov 03 14:12:05.748832 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.748801 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4z4z\" (UniqueName: \"kubernetes.io/projected/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-kube-api-access-h4z4z\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:12:05.749072 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.749047 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq4pb\" (UniqueName: \"kubernetes.io/projected/40e20104-e97c-41b2-8530-abb2dec148a2-kube-api-access-kq4pb\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.750869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.749058 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dch9b\" (UniqueName: \"kubernetes.io/projected/6f1c105f-a7fa-4aeb-b8de-a413f5def9bf-kube-api-access-dch9b\") pod \"network-check-target-phmgc\" (UID: \"6f1c105f-a7fa-4aeb-b8de-a413f5def9bf\") " pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:12:05.750869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.749213 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkjnc\" (UniqueName: \"kubernetes.io/projected/05a67278-34b4-462d-b4cb-9250800bf3db-kube-api-access-wkjnc\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.750869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.749343 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlv2\" (UniqueName: \"kubernetes.io/projected/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-kube-api-access-5dlv2\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.750869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.749486 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wbc\" (UniqueName: \"kubernetes.io/projected/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-kube-api-access-f2wbc\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.750869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.749523 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d4lt\" (UniqueName: \"kubernetes.io/projected/b586d272-c9c0-455f-aec9-333ae6d0c69a-kube-api-access-7d4lt\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:12:05.750869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.749760 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7d5k\" (UniqueName: \"kubernetes.io/projected/0694b8ee-14f6-495d-a662-8f06bdfe1c66-kube-api-access-m7d5k\") pod \"downloads-6dd789c8b7-dzpfq\" (UID: \"0694b8ee-14f6-495d-a662-8f06bdfe1c66\") " pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:12:05.750869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.750068 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2q54\" (UniqueName: \"kubernetes.io/projected/977c941d-36f2-4c16-8396-95bdce627a74-kube-api-access-g2q54\") pod \"dns-default-j6lgv\" (UID: \"977c941d-36f2-4c16-8396-95bdce627a74\") " pod="openshift-dns/dns-default-j6lgv" Nov 03 14:12:05.826236 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826153 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-etc-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.826236 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826197 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.826517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826270 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.826517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826279 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-etc-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.826517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826300 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-host\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.826517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826343 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-host\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.826517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826357 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-var-lib-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.826517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826383 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-ovn\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.826517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826410 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-var-lib-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.826517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826412 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edb1d4a1-0656-4417-8b7d-86a68db196ee-host\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:12:05.826517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826455 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edb1d4a1-0656-4417-8b7d-86a68db196ee-host\") pod \"node-ca-dlwbh\" (UID: \"edb1d4a1-0656-4417-8b7d-86a68db196ee\") " pod="openshift-image-registry/node-ca-dlwbh" Nov 03 14:12:05.826517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826476 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-ovn\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.826517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826485 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-sys\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826526 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-host-slash\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826555 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-kubelet-config\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826556 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-sys\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826581 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826596 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6b9aee7-f7fb-4a15-8527-9ad5bda2523d-host-slash\") pod \"iptables-alerter-9pdj9\" (UID: \"d6b9aee7-f7fb-4a15-8527-9ad5bda2523d\") " pod="openshift-network-operator/iptables-alerter-9pdj9" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826633 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-kubelet-config\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826633 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-openvswitch\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826653 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-crio-socket\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826684 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-socket-dir-parent\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826710 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-kubelet\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826735 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-cnibin\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826749 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f824baea-c6ee-409c-a1d3-e4f8b02bc6c5-crio-socket\") pod \"insights-runtime-extractor-wqbbs\" (UID: \"f824baea-c6ee-409c-a1d3-e4f8b02bc6c5\") " pod="openshift-insights/insights-runtime-extractor-wqbbs" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826767 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-socket-dir-parent\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826791 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-socket-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826816 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-kubelet\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826820 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-cnibin\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826828 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-hostroot\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826859 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-registration-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826878 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-socket-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826883 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-root\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826916 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-multus-certs\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826931 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-root\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826944 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-device-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826965 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-multus-certs\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826971 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-dbus\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826994 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-registration-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827007 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-kubelet\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827020 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc214996-6ee2-4bf3-a2cd-49cd4eceaa09-dbus\") pod \"global-pull-secret-syncer-tjsp9\" (UID: \"fc214996-6ee2-4bf3-a2cd-49cd4eceaa09\") " pod="kube-system/global-pull-secret-syncer-tjsp9" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.826933 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-hostroot\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827032 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-lib-modules\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827042 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-device-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827069 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-sys-fs\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827092 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-kubelet\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827125 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-netns\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827097 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-netns\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827139 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-sys-fs\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827147 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-lib-modules\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.827133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827162 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-etc-kubernetes\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827190 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-systemd-units\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827216 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-run-netns\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827257 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-systemd-units\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827259 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-run-netns\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827277 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-var-lib-kubelet\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827294 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-etc-kubernetes\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827310 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-cnibin\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827333 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827340 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-var-lib-kubelet\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827359 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-kubernetes\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827414 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-cnibin\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827422 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827456 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-log-socket\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827465 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-kubernetes\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827484 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-cni-netd\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827512 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-cni-bin\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827519 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-log-socket\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827538 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-cni-multus\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827553 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-cni-netd\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827567 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827590 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-modprobe-d\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827595 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-cni-bin\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827630 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysctl-conf\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827648 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-cni-bin\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827666 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-systemd\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827672 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827689 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-system-cni-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827712 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-system-cni-dir\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827715 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-var-lib-cni-multus\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827740 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827756 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-cni-bin\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827763 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-sys\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827813 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-sys\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827858 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-run-systemd\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827860 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-modprobe-d\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827879 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-slash\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827909 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-system-cni-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827906 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysctl-d\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827930 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-systemd\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827952 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-wtmp\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827992 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-cni-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828038 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b586d272-c9c0-455f-aec9-333ae6d0c69a-hosts-file\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828064 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-run\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828087 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-conf-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828114 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysctl-conf\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828123 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828159 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-k8s-cni-cncf-io\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828183 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-node-log\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828189 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e20104-e97c-41b2-8530-abb2dec148a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6ppcs\" (UID: \"40e20104-e97c-41b2-8530-abb2dec148a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ppcs" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828208 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysconfig\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828226 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05a67278-34b4-462d-b4cb-9250800bf3db-system-cni-dir\") pod \"multus-additional-cni-plugins-99kd6\" (UID: \"05a67278-34b4-462d-b4cb-9250800bf3db\") " pod="openshift-multus/multus-additional-cni-plugins-99kd6" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828281 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-slash\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828305 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-cni-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828315 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-host-run-k8s-cni-cncf-io\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828353 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-node-log\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828398 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysconfig\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828398 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b57fa4d4-1e9d-4a56-aba9-c6314ff12aea-node-exporter-wtmp\") pod \"node-exporter-zbs97\" (UID: \"b57fa4d4-1e9d-4a56-aba9-c6314ff12aea\") " pod="openshift-monitoring/node-exporter-zbs97" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.827991 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-sysctl-d\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828437 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b586d272-c9c0-455f-aec9-333ae6d0c69a-hosts-file\") pod \"node-resolver-qnhkx\" (UID: \"b586d272-c9c0-455f-aec9-333ae6d0c69a\") " pod="openshift-dns/node-resolver-qnhkx" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828479 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2a85509-0811-4aaf-a86d-b41e5ae5a9cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-7j9ks\" (UID: \"f2a85509-0811-4aaf-a86d-b41e5ae5a9cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828486 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-etc-systemd\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828527 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b00819ff-6b3d-4e36-995b-2d671e0dc9f6-run\") pod \"tuned-747sw\" (UID: \"b00819ff-6b3d-4e36-995b-2d671e0dc9f6\") " pod="openshift-cluster-node-tuning-operator/tuned-747sw" Nov 03 14:12:05.828983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.828548 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d153593a-c25f-4bef-acca-ea7e50b32c87-multus-conf-dir\") pod \"multus-vlblr\" (UID: \"d153593a-c25f-4bef-acca-ea7e50b32c87\") " pod="openshift-multus/multus-vlblr" Nov 03 14:12:05.937369 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:05.937338 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c65c5d7-j9sv7"] Nov 03 14:12:07.355679 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:07.355642 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:07.692501 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:07.692468 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j6lgv" Nov 03 14:12:08.812600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:08.812570 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-frkjs" Nov 03 14:12:12.479039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:12.479010 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:12:12.481148 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:12.481119 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-dzpfq" Nov 03 14:12:12.582200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:12.582174 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:12:12.891343 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:12.891273 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:13.415129 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:13.415040 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:13.419681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:13.419661 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:13.780547 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:13.780515 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:13.979420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:13.979393 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:13.979602 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:13.979590 6389 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 14:12:19.274658 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.274598 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-d56c57f9b-fjg7p"] Nov 03 14:12:19.301819 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.301788 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d56c57f9b-fjg7p"] Nov 03 14:12:19.302283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.302257 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.437774 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.437744 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fbb10eb-9066-40b9-84c6-315b93015a50-console-serving-cert\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.438020 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.437782 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fbb10eb-9066-40b9-84c6-315b93015a50-oauth-serving-cert\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.438020 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.437802 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fbb10eb-9066-40b9-84c6-315b93015a50-console-config\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.438020 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.437876 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fbb10eb-9066-40b9-84c6-315b93015a50-trusted-ca-bundle\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.438020 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.437959 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbb10eb-9066-40b9-84c6-315b93015a50-service-ca\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.438020 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.438006 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fbb10eb-9066-40b9-84c6-315b93015a50-console-oauth-config\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.438276 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.438097 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fbb10eb-9066-40b9-84c6-315b93015a50-tmp\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.438276 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.438134 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbj2s\" (UniqueName: \"kubernetes.io/projected/4fbb10eb-9066-40b9-84c6-315b93015a50-kube-api-access-kbj2s\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.539465 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.539385 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbj2s\" (UniqueName: \"kubernetes.io/projected/4fbb10eb-9066-40b9-84c6-315b93015a50-kube-api-access-kbj2s\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.539465 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.539440 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fbb10eb-9066-40b9-84c6-315b93015a50-console-serving-cert\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.539745 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.539478 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fbb10eb-9066-40b9-84c6-315b93015a50-oauth-serving-cert\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.539745 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.539509 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fbb10eb-9066-40b9-84c6-315b93015a50-console-config\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.539745 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.539533 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fbb10eb-9066-40b9-84c6-315b93015a50-trusted-ca-bundle\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.539745 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.539589 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbb10eb-9066-40b9-84c6-315b93015a50-service-ca\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.539745 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.539660 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fbb10eb-9066-40b9-84c6-315b93015a50-console-oauth-config\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.539745 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.539710 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fbb10eb-9066-40b9-84c6-315b93015a50-tmp\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.540031 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.539994 6389 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 14:12:19.540233 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.540213 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fbb10eb-9066-40b9-84c6-315b93015a50-tmp\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.540510 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.540486 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbb10eb-9066-40b9-84c6-315b93015a50-service-ca\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.540597 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.540578 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fbb10eb-9066-40b9-84c6-315b93015a50-console-config\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.540740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.540721 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fbb10eb-9066-40b9-84c6-315b93015a50-trusted-ca-bundle\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.540832 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.540809 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fbb10eb-9066-40b9-84c6-315b93015a50-oauth-serving-cert\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.542366 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.542339 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fbb10eb-9066-40b9-84c6-315b93015a50-console-oauth-config\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.542446 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.542430 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fbb10eb-9066-40b9-84c6-315b93015a50-console-serving-cert\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.559526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.559498 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbj2s\" (UniqueName: \"kubernetes.io/projected/4fbb10eb-9066-40b9-84c6-315b93015a50-kube-api-access-kbj2s\") pod \"console-d56c57f9b-fjg7p\" (UID: \"4fbb10eb-9066-40b9-84c6-315b93015a50\") " pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.627748 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.627709 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:19.950158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:19.949995 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d56c57f9b-fjg7p"] Nov 03 14:12:19.952909 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:12:19.952880 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbb10eb_9066_40b9_84c6_315b93015a50.slice/crio-c073953ecae933375658e4309d9c0a416963e8841b58cc5e2527c549d25dae63 WatchSource:0}: Error finding container c073953ecae933375658e4309d9c0a416963e8841b58cc5e2527c549d25dae63: Status 404 returned error can't find the container with id c073953ecae933375658e4309d9c0a416963e8841b58cc5e2527c549d25dae63 Nov 03 14:12:20.769332 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:20.769296 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d56c57f9b-fjg7p" event={"ID":"4fbb10eb-9066-40b9-84c6-315b93015a50","Type":"ContainerStarted","Data":"cea17e19f2626d22066ddd159d89925fe327b09098a329393f4e0eecae33c2ff"} Nov 03 14:12:20.769820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:20.769344 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d56c57f9b-fjg7p" event={"ID":"4fbb10eb-9066-40b9-84c6-315b93015a50","Type":"ContainerStarted","Data":"c073953ecae933375658e4309d9c0a416963e8841b58cc5e2527c549d25dae63"} Nov 03 14:12:20.812996 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:20.812939 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-d56c57f9b-fjg7p" podStartSLOduration=1.812927459 podStartE2EDuration="1.812927459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:20.812896276 +0000 UTC m=+16.371532761" watchObservedRunningTime="2026-11-03 14:12:20.812927459 +0000 UTC m=+16.371564008" Nov 03 14:12:23.786062 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:23.786037 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:27.644249 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:27.644213 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7j9ks" Nov 03 14:12:28.375018 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:28.374979 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:12:29.628574 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:29.628539 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:29.628986 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:29.628597 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:29.633171 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:29.633153 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:29.804092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:29.804064 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d56c57f9b-fjg7p" Nov 03 14:12:29.879576 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:29.879505 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c959ffc98-lphmd"] Nov 03 14:12:31.742217 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:31.742153 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-84c65c5d7-j9sv7" podUID="61d8c282-03cc-4a20-a999-6cda0b14c87e" containerName="console" containerID="cri-o://4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920" gracePeriod=15 Nov 03 14:12:32.017806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.017744 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:32.030710 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.030681 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-service-ca\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-service-ca\") pod \"61d8c282-03cc-4a20-a999-6cda0b14c87e\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " Nov 03 14:12:32.030845 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.030729 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-config\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-config\") pod \"61d8c282-03cc-4a20-a999-6cda0b14c87e\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " Nov 03 14:12:32.030845 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.030770 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-oauth-serving-cert\") pod \"61d8c282-03cc-4a20-a999-6cda0b14c87e\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " Nov 03 14:12:32.030845 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.030826 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-oauth-config\") pod \"61d8c282-03cc-4a20-a999-6cda0b14c87e\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " Nov 03 14:12:32.030975 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.030853 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d8c282-03cc-4a20-a999-6cda0b14c87e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/61d8c282-03cc-4a20-a999-6cda0b14c87e-tmp\") pod \"61d8c282-03cc-4a20-a999-6cda0b14c87e\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " Nov 03 14:12:32.030975 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.030888 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-serving-cert\") pod \"61d8c282-03cc-4a20-a999-6cda0b14c87e\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " Nov 03 14:12:32.030975 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.030916 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d8c282-03cc-4a20-a999-6cda0b14c87e-kube-api-access-7j5qj\" (UniqueName: \"kubernetes.io/projected/61d8c282-03cc-4a20-a999-6cda0b14c87e-kube-api-access-7j5qj\") pod \"61d8c282-03cc-4a20-a999-6cda0b14c87e\" (UID: \"61d8c282-03cc-4a20-a999-6cda0b14c87e\") " Nov 03 14:12:32.031429 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.031196 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d8c282-03cc-4a20-a999-6cda0b14c87e-tmp" pod "61d8c282-03cc-4a20-a999-6cda0b14c87e" (UID: "61d8c282-03cc-4a20-a999-6cda0b14c87e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:32.031429 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.031238 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-config" pod "61d8c282-03cc-4a20-a999-6cda0b14c87e" (UID: "61d8c282-03cc-4a20-a999-6cda0b14c87e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:32.031429 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.031214 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-service-ca" pod "61d8c282-03cc-4a20-a999-6cda0b14c87e" (UID: "61d8c282-03cc-4a20-a999-6cda0b14c87e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:32.031429 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.031251 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-oauth-serving-cert" pod "61d8c282-03cc-4a20-a999-6cda0b14c87e" (UID: "61d8c282-03cc-4a20-a999-6cda0b14c87e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:32.031429 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.031354 6389 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-config\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:32.031429 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.031376 6389 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-oauth-serving-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:32.031429 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.031390 6389 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61d8c282-03cc-4a20-a999-6cda0b14c87e-tmp\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:32.031429 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.031403 6389 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61d8c282-03cc-4a20-a999-6cda0b14c87e-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:32.033263 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.033244 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-oauth-config" pod "61d8c282-03cc-4a20-a999-6cda0b14c87e" (UID: "61d8c282-03cc-4a20-a999-6cda0b14c87e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:12:32.033435 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.033414 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d8c282-03cc-4a20-a999-6cda0b14c87e-kube-api-access-7j5qj" pod "61d8c282-03cc-4a20-a999-6cda0b14c87e" (UID: "61d8c282-03cc-4a20-a999-6cda0b14c87e"). InnerVolumeSpecName "kube-api-access-7j5qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:32.033772 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.033747 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-serving-cert" pod "61d8c282-03cc-4a20-a999-6cda0b14c87e" (UID: "61d8c282-03cc-4a20-a999-6cda0b14c87e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:12:32.132061 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.132027 6389 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-oauth-config\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:32.132061 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.132054 6389 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61d8c282-03cc-4a20-a999-6cda0b14c87e-console-serving-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:32.132061 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.132065 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j5qj\" (UniqueName: \"kubernetes.io/projected/61d8c282-03cc-4a20-a999-6cda0b14c87e-kube-api-access-7j5qj\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:32.584585 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.584556 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-phmgc" Nov 03 14:12:32.810353 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.810315 6389 generic.go:356] "Generic (PLEG): container finished" podID="61d8c282-03cc-4a20-a999-6cda0b14c87e" containerID="4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920" exitCode=0 Nov 03 14:12:32.810790 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.810365 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c65c5d7-j9sv7" Nov 03 14:12:32.810790 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.810386 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c65c5d7-j9sv7" event={"ID":"61d8c282-03cc-4a20-a999-6cda0b14c87e","Type":"ContainerDied","Data":"4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920"} Nov 03 14:12:32.810790 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.810432 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c65c5d7-j9sv7" event={"ID":"61d8c282-03cc-4a20-a999-6cda0b14c87e","Type":"ContainerDied","Data":"d9a77ad916f5ee90d19fd297849f1a6e4f9be2ee781e2be5d061d74572af8362"} Nov 03 14:12:32.810790 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.810453 6389 scope.go:122] "RemoveContainer" containerID="4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920" Nov 03 14:12:32.819081 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.819064 6389 scope.go:122] "RemoveContainer" containerID="4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920" Nov 03 14:12:32.819364 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:32.819346 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920\": container with ID starting with 4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920 not found: ID does not exist" containerID="4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920" Nov 03 14:12:32.819415 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.819373 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920"} err="failed to get container status \"4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920\": rpc error: code = NotFound desc = could not find container \"4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920\": container with ID starting with 4d70e05fea9234a4dc55242cd751e7d4c1e96c41d807eed3160e1c607d4ba920 not found: ID does not exist" Nov 03 14:12:32.860172 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.860101 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c65c5d7-j9sv7"] Nov 03 14:12:32.871587 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:32.871549 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84c65c5d7-j9sv7"] Nov 03 14:12:34.614981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:34.614948 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d8c282-03cc-4a20-a999-6cda0b14c87e" path="/var/lib/kubelet/pods/61d8c282-03cc-4a20-a999-6cda0b14c87e/volumes" Nov 03 14:12:54.899157 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:54.899088 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c959ffc98-lphmd" podUID="a22219af-1262-42a9-8cbe-8e6323e4375b" containerName="console" containerID="cri-o://0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3" gracePeriod=15 Nov 03 14:12:55.143585 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.143561 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:55.206419 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.206387 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22219af-1262-42a9-8cbe-8e6323e4375b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a22219af-1262-42a9-8cbe-8e6323e4375b-tmp\") pod \"a22219af-1262-42a9-8cbe-8e6323e4375b\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " Nov 03 14:12:55.206581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.206437 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-oauth-config\") pod \"a22219af-1262-42a9-8cbe-8e6323e4375b\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " Nov 03 14:12:55.206581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.206467 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-serving-cert\") pod \"a22219af-1262-42a9-8cbe-8e6323e4375b\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " Nov 03 14:12:55.206581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.206490 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-trusted-ca-bundle\") pod \"a22219af-1262-42a9-8cbe-8e6323e4375b\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " Nov 03 14:12:55.206581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.206537 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-console-config\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-console-config\") pod \"a22219af-1262-42a9-8cbe-8e6323e4375b\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " Nov 03 14:12:55.206841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.206580 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22219af-1262-42a9-8cbe-8e6323e4375b-kube-api-access-wd8sm\" (UniqueName: \"kubernetes.io/projected/a22219af-1262-42a9-8cbe-8e6323e4375b-kube-api-access-wd8sm\") pod \"a22219af-1262-42a9-8cbe-8e6323e4375b\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " Nov 03 14:12:55.206841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.206648 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-oauth-serving-cert\") pod \"a22219af-1262-42a9-8cbe-8e6323e4375b\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " Nov 03 14:12:55.206841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.206688 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-service-ca\") pod \"a22219af-1262-42a9-8cbe-8e6323e4375b\" (UID: \"a22219af-1262-42a9-8cbe-8e6323e4375b\") " Nov 03 14:12:55.206841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.206702 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22219af-1262-42a9-8cbe-8e6323e4375b-tmp" pod "a22219af-1262-42a9-8cbe-8e6323e4375b" (UID: "a22219af-1262-42a9-8cbe-8e6323e4375b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:55.207042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.206901 6389 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a22219af-1262-42a9-8cbe-8e6323e4375b-tmp\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:55.207095 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.207052 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-console-config" pod "a22219af-1262-42a9-8cbe-8e6323e4375b" (UID: "a22219af-1262-42a9-8cbe-8e6323e4375b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:55.207095 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.207062 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-oauth-serving-cert" pod "a22219af-1262-42a9-8cbe-8e6323e4375b" (UID: "a22219af-1262-42a9-8cbe-8e6323e4375b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:55.207358 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.207335 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-trusted-ca-bundle" pod "a22219af-1262-42a9-8cbe-8e6323e4375b" (UID: "a22219af-1262-42a9-8cbe-8e6323e4375b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:55.207472 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.207375 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-service-ca" pod "a22219af-1262-42a9-8cbe-8e6323e4375b" (UID: "a22219af-1262-42a9-8cbe-8e6323e4375b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:55.208901 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.208873 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-serving-cert" pod "a22219af-1262-42a9-8cbe-8e6323e4375b" (UID: "a22219af-1262-42a9-8cbe-8e6323e4375b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:12:55.208962 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.208916 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22219af-1262-42a9-8cbe-8e6323e4375b-kube-api-access-wd8sm" pod "a22219af-1262-42a9-8cbe-8e6323e4375b" (UID: "a22219af-1262-42a9-8cbe-8e6323e4375b"). InnerVolumeSpecName "kube-api-access-wd8sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:55.208962 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.208933 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-oauth-config" pod "a22219af-1262-42a9-8cbe-8e6323e4375b" (UID: "a22219af-1262-42a9-8cbe-8e6323e4375b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:12:55.307276 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.307228 6389 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-oauth-config\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:55.307276 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.307269 6389 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a22219af-1262-42a9-8cbe-8e6323e4375b-console-serving-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:55.307276 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.307279 6389 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-trusted-ca-bundle\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:55.307276 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.307290 6389 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-console-config\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:55.307537 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.307300 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd8sm\" (UniqueName: \"kubernetes.io/projected/a22219af-1262-42a9-8cbe-8e6323e4375b-kube-api-access-wd8sm\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:55.307537 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.307335 6389 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-oauth-serving-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:55.307537 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.307345 6389 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a22219af-1262-42a9-8cbe-8e6323e4375b-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:12:55.902219 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.902186 6389 generic.go:356] "Generic (PLEG): container finished" podID="a22219af-1262-42a9-8cbe-8e6323e4375b" containerID="0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3" exitCode=0 Nov 03 14:12:55.902657 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.902249 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c959ffc98-lphmd" Nov 03 14:12:55.902657 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.902268 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c959ffc98-lphmd" event={"ID":"a22219af-1262-42a9-8cbe-8e6323e4375b","Type":"ContainerDied","Data":"0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3"} Nov 03 14:12:55.902657 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.902309 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c959ffc98-lphmd" event={"ID":"a22219af-1262-42a9-8cbe-8e6323e4375b","Type":"ContainerDied","Data":"fafcf58717120bce6dc686e7b602493ee4ad8879d233785ca2b2bf90c909e84e"} Nov 03 14:12:55.902657 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.902328 6389 scope.go:122] "RemoveContainer" containerID="0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3" Nov 03 14:12:55.910824 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.910805 6389 scope.go:122] "RemoveContainer" containerID="0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3" Nov 03 14:12:55.911056 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:55.911042 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3\": container with ID starting with 0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3 not found: ID does not exist" containerID="0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3" Nov 03 14:12:55.911111 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.911065 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3"} err="failed to get container status \"0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3\": rpc error: code = NotFound desc = could not find container \"0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3\": container with ID starting with 0fec21969898270e601e7092ed7940ceb104fb14b742eb93bc87cb45f61b31a3 not found: ID does not exist" Nov 03 14:12:55.953978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.953946 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c959ffc98-lphmd"] Nov 03 14:12:55.965144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:55.965116 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c959ffc98-lphmd"] Nov 03 14:12:56.614870 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:12:56.614838 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22219af-1262-42a9-8cbe-8e6323e4375b" path="/var/lib/kubelet/pods/a22219af-1262-42a9-8cbe-8e6323e4375b/volumes" Nov 03 14:12:58.372348 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:12:58.372313 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:13:28.372456 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:13:28.372417 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:13:58.372181 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:13:58.372098 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:14:28.371963 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:14:28.371927 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:14:58.371830 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:14:58.371792 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:15:11.699065 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.699029 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn"] Nov 03 14:15:11.699590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.699368 6389 cpu_manager.go:414] "containerMap: removing container" podUID="61d8c282-03cc-4a20-a999-6cda0b14c87e" containerName="console" Nov 03 14:15:11.699590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.699385 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d8c282-03cc-4a20-a999-6cda0b14c87e" containerName="console" Nov 03 14:15:11.699590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.699403 6389 cpu_manager.go:414] "containerMap: removing container" podUID="a22219af-1262-42a9-8cbe-8e6323e4375b" containerName="console" Nov 03 14:15:11.699590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.699411 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22219af-1262-42a9-8cbe-8e6323e4375b" containerName="console" Nov 03 14:15:11.699590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.699469 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d8c282-03cc-4a20-a999-6cda0b14c87e" containerName="console" Nov 03 14:15:11.699590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.699477 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22219af-1262-42a9-8cbe-8e6323e4375b" containerName="console" Nov 03 14:15:11.701498 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.701477 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:11.729158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.729131 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 14:15:11.729158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.729131 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 14:15:11.733486 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.733469 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn"] Nov 03 14:15:11.750854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.750834 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2zwzh\"" Nov 03 14:15:11.794959 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.794929 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn\" (UID: \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:11.795105 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.794966 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmr2x\" (UniqueName: \"kubernetes.io/projected/869d7b42-442f-4de3-bcdd-afb8b29a51ce-kube-api-access-qmr2x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn\" (UID: \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:11.795105 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.794995 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn\" (UID: \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:11.818788 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.818759 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2"] Nov 03 14:15:11.820960 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.820941 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:11.836786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.836767 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-sfbzt\"" Nov 03 14:15:11.843360 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.843340 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2"] Nov 03 14:15:11.895743 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.895709 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn\" (UID: \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:11.895743 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.895747 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2\" (UID: \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:11.895939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.895787 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2\" (UID: \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:11.895939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.895806 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svps7\" (UniqueName: \"kubernetes.io/projected/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-kube-api-access-svps7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2\" (UID: \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:11.895939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.895837 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn\" (UID: \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:11.895939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.895857 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmr2x\" (UniqueName: \"kubernetes.io/projected/869d7b42-442f-4de3-bcdd-afb8b29a51ce-kube-api-access-qmr2x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn\" (UID: \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:11.896185 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.896164 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn\" (UID: \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:11.896221 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.896189 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn\" (UID: \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:11.917432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.917390 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmr2x\" (UniqueName: \"kubernetes.io/projected/869d7b42-442f-4de3-bcdd-afb8b29a51ce-kube-api-access-qmr2x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn\" (UID: \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:11.996661 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.996562 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2\" (UID: \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:11.996661 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.996646 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2\" (UID: \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:11.996848 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.996667 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svps7\" (UniqueName: \"kubernetes.io/projected/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-kube-api-access-svps7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2\" (UID: \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:11.997039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.997016 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2\" (UID: \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:11.997075 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:11.997021 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2\" (UID: \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:12.013820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.013791 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:12.017943 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.017915 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svps7\" (UniqueName: \"kubernetes.io/projected/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-kube-api-access-svps7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2\" (UID: \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:12.068427 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.068401 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk"] Nov 03 14:15:12.075744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.074844 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.090684 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.090664 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 14:15:12.090684 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.090684 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 14:15:12.090940 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.090907 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 14:15:12.091066 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.090975 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dmczr\"" Nov 03 14:15:12.091066 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.090990 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 14:15:12.091066 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.091037 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 14:15:12.091066 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.091047 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 14:15:12.096262 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.096237 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk"] Nov 03 14:15:12.133377 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.133334 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:12.198143 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.198115 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.198319 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.198153 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.198319 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.198181 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.198319 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.198253 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.198482 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.198370 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhljv\" (UniqueName: \"kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-kube-api-access-vhljv\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.198482 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.198393 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f82a42-a4ff-43dc-96c5-56135f30576a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.198482 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.198419 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.299518 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.299438 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f82a42-a4ff-43dc-96c5-56135f30576a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.299518 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.299481 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.299715 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.299641 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.299715 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.299694 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.299813 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.299757 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.299813 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.299806 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.299903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.299845 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhljv\" (UniqueName: \"kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-kube-api-access-vhljv\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.300440 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.300395 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.302001 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.301974 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.302001 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.301992 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f82a42-a4ff-43dc-96c5-56135f30576a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.302151 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.302094 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.302151 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.302126 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.314602 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.314494 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn"] Nov 03 14:15:12.321804 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:15:12.321767 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869d7b42_442f_4de3_bcdd_afb8b29a51ce.slice/crio-ba0ab28962bcfcd4509376ab5f00c3f554a4dcad7c57bb2be068c0e0c050ba6c WatchSource:0}: Error finding container ba0ab28962bcfcd4509376ab5f00c3f554a4dcad7c57bb2be068c0e0c050ba6c: Status 404 returned error can't find the container with id ba0ab28962bcfcd4509376ab5f00c3f554a4dcad7c57bb2be068c0e0c050ba6c Nov 03 14:15:12.322666 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.322499 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.322781 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.322759 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhljv\" (UniqueName: \"kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-kube-api-access-vhljv\") pod \"istiod-openshift-gateway-7876bfcb74-mf4wk\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.324460 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.324443 6389 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:15:12.337195 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.337171 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" event={"ID":"869d7b42-442f-4de3-bcdd-afb8b29a51ce","Type":"ContainerStarted","Data":"ba0ab28962bcfcd4509376ab5f00c3f554a4dcad7c57bb2be068c0e0c050ba6c"} Nov 03 14:15:12.386944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.386917 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:12.433904 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.433872 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2"] Nov 03 14:15:12.442738 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:15:12.440595 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c6ef7f_5968_4f97_994c_964c2ea28b4a.slice/crio-ed9ce6c24f124717377e4584848a5961a2ab4ad2d6d22a7c109b76ad8fdc01fd WatchSource:0}: Error finding container ed9ce6c24f124717377e4584848a5961a2ab4ad2d6d22a7c109b76ad8fdc01fd: Status 404 returned error can't find the container with id ed9ce6c24f124717377e4584848a5961a2ab4ad2d6d22a7c109b76ad8fdc01fd Nov 03 14:15:12.687757 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:12.687729 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk"] Nov 03 14:15:12.690575 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:15:12.690542 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f82a42_a4ff_43dc_96c5_56135f30576a.slice/crio-6ea80635f4575e0e98d4c3d63ef0a2fb903c9eeaeb253a71b70f82af85be2782 WatchSource:0}: Error finding container 6ea80635f4575e0e98d4c3d63ef0a2fb903c9eeaeb253a71b70f82af85be2782: Status 404 returned error can't find the container with id 6ea80635f4575e0e98d4c3d63ef0a2fb903c9eeaeb253a71b70f82af85be2782 Nov 03 14:15:13.344088 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:13.343885 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" event={"ID":"90f82a42-a4ff-43dc-96c5-56135f30576a","Type":"ContainerStarted","Data":"6ea80635f4575e0e98d4c3d63ef0a2fb903c9eeaeb253a71b70f82af85be2782"} Nov 03 14:15:13.347581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:13.347547 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" event={"ID":"e3c6ef7f-5968-4f97-994c-964c2ea28b4a","Type":"ContainerStarted","Data":"ed9ce6c24f124717377e4584848a5961a2ab4ad2d6d22a7c109b76ad8fdc01fd"} Nov 03 14:15:15.768841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:15.768815 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:15:15.769073 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:15.768875 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:15:16.362561 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:16.362524 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" event={"ID":"90f82a42-a4ff-43dc-96c5-56135f30576a","Type":"ContainerStarted","Data":"458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61"} Nov 03 14:15:16.362760 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:16.362718 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:16.364551 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:16.364526 6389 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-mf4wk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:15:16.364716 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:16.364592 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" podUID="90f82a42-a4ff-43dc-96c5-56135f30576a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:15:16.429162 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:16.429106 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" podStartSLOduration=1.353712515 podStartE2EDuration="4.429090382s" totalImagesPullingTime="3.075377867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:16.428836974 +0000 UTC m=+191.987473459" watchObservedRunningTime="2026-11-03 14:15:16.429090382 +0000 UTC m=+191.987726868" Nov 03 14:15:17.368188 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:17.367954 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:15:20.379487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:20.379454 6389 generic.go:356] "Generic (PLEG): container finished" podID="e3c6ef7f-5968-4f97-994c-964c2ea28b4a" containerID="3cd7ff2e250d0544375798f1fa0b20e17dd0a9964147f5a546da131c0c45a012" exitCode=0 Nov 03 14:15:20.380009 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:20.379542 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" event={"ID":"e3c6ef7f-5968-4f97-994c-964c2ea28b4a","Type":"ContainerDied","Data":"3cd7ff2e250d0544375798f1fa0b20e17dd0a9964147f5a546da131c0c45a012"} Nov 03 14:15:20.381240 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:20.381221 6389 generic.go:356] "Generic (PLEG): container finished" podID="869d7b42-442f-4de3-bcdd-afb8b29a51ce" containerID="84b1ba8e30fce7f83e925d30580e0adc1ba062b78278d125932930319633e20c" exitCode=0 Nov 03 14:15:20.381314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:20.381258 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" event={"ID":"869d7b42-442f-4de3-bcdd-afb8b29a51ce","Type":"ContainerDied","Data":"84b1ba8e30fce7f83e925d30580e0adc1ba062b78278d125932930319633e20c"} Nov 03 14:15:21.388188 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:21.388109 6389 generic.go:356] "Generic (PLEG): container finished" podID="e3c6ef7f-5968-4f97-994c-964c2ea28b4a" containerID="9d8a90bc264e46e0b71c568ceb342553729c777f21622086a6a0c152fd131ef3" exitCode=0 Nov 03 14:15:21.388188 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:21.388165 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" event={"ID":"e3c6ef7f-5968-4f97-994c-964c2ea28b4a","Type":"ContainerDied","Data":"9d8a90bc264e46e0b71c568ceb342553729c777f21622086a6a0c152fd131ef3"} Nov 03 14:15:23.398756 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:23.398685 6389 generic.go:356] "Generic (PLEG): container finished" podID="869d7b42-442f-4de3-bcdd-afb8b29a51ce" containerID="932d08520728f0996e212c47626e60818f2f0a4b84836ed6dc86812f1180d371" exitCode=0 Nov 03 14:15:23.399230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:23.398707 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" event={"ID":"869d7b42-442f-4de3-bcdd-afb8b29a51ce","Type":"ContainerDied","Data":"932d08520728f0996e212c47626e60818f2f0a4b84836ed6dc86812f1180d371"} Nov 03 14:15:28.382366 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:15:28.382335 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:15:28.422319 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:28.422286 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" event={"ID":"e3c6ef7f-5968-4f97-994c-964c2ea28b4a","Type":"ContainerStarted","Data":"0040d9aa384df9aa79add14b0b4212145701e43c195c91eb5e29390a18574c5f"} Nov 03 14:15:28.466198 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:28.466142 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" podStartSLOduration=2.7146917889999997 podStartE2EDuration="17.466128274s" totalImagesPullingTime="14.751436485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:28.465574821 +0000 UTC m=+204.024211304" watchObservedRunningTime="2026-11-03 14:15:28.466128274 +0000 UTC m=+204.024764818" Nov 03 14:15:29.429860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:29.429819 6389 generic.go:356] "Generic (PLEG): container finished" podID="e3c6ef7f-5968-4f97-994c-964c2ea28b4a" containerID="0040d9aa384df9aa79add14b0b4212145701e43c195c91eb5e29390a18574c5f" exitCode=0 Nov 03 14:15:29.430321 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:29.429881 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" event={"ID":"e3c6ef7f-5968-4f97-994c-964c2ea28b4a","Type":"ContainerDied","Data":"0040d9aa384df9aa79add14b0b4212145701e43c195c91eb5e29390a18574c5f"} Nov 03 14:15:30.435941 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:30.435909 6389 generic.go:356] "Generic (PLEG): container finished" podID="869d7b42-442f-4de3-bcdd-afb8b29a51ce" containerID="530226d8f83e09489d986023300645ddd400756ea3943ad35d774315d470eceb" exitCode=0 Nov 03 14:15:30.436322 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:30.435995 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" event={"ID":"869d7b42-442f-4de3-bcdd-afb8b29a51ce","Type":"ContainerDied","Data":"530226d8f83e09489d986023300645ddd400756ea3943ad35d774315d470eceb"} Nov 03 14:15:30.559238 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:30.559213 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:30.565932 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:30.565914 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-util\" (UniqueName: \"kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-util\") pod \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\" (UID: \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\") " Nov 03 14:15:30.566011 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:30.565958 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-kube-api-access-svps7\" (UniqueName: \"kubernetes.io/projected/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-kube-api-access-svps7\") pod \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\" (UID: \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\") " Nov 03 14:15:30.566045 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:30.566012 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-bundle\") pod \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\" (UID: \"e3c6ef7f-5968-4f97-994c-964c2ea28b4a\") " Nov 03 14:15:30.566621 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:30.566579 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-bundle" pod "e3c6ef7f-5968-4f97-994c-964c2ea28b4a" (UID: "e3c6ef7f-5968-4f97-994c-964c2ea28b4a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:30.568311 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:30.568286 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-kube-api-access-svps7" pod "e3c6ef7f-5968-4f97-994c-964c2ea28b4a" (UID: "e3c6ef7f-5968-4f97-994c-964c2ea28b4a"). InnerVolumeSpecName "kube-api-access-svps7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:30.570311 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:30.570287 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-util" pod "e3c6ef7f-5968-4f97-994c-964c2ea28b4a" (UID: "e3c6ef7f-5968-4f97-994c-964c2ea28b4a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:30.667336 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:30.667309 6389 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-util\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:15:30.667336 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:30.667334 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svps7\" (UniqueName: \"kubernetes.io/projected/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-kube-api-access-svps7\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:15:30.667522 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:30.667346 6389 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3c6ef7f-5968-4f97-994c-964c2ea28b4a-bundle\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:15:31.440969 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.440941 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" Nov 03 14:15:31.441461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.440969 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714zwd2" event={"ID":"e3c6ef7f-5968-4f97-994c-964c2ea28b4a","Type":"ContainerDied","Data":"ed9ce6c24f124717377e4584848a5961a2ab4ad2d6d22a7c109b76ad8fdc01fd"} Nov 03 14:15:31.441461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.441014 6389 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed9ce6c24f124717377e4584848a5961a2ab4ad2d6d22a7c109b76ad8fdc01fd" Nov 03 14:15:31.564984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.564957 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:31.574482 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.574459 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-util\" (UniqueName: \"kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-util\") pod \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\" (UID: \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\") " Nov 03 14:15:31.574646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.574515 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-bundle\" (UniqueName: \"kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-bundle\") pod \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\" (UID: \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\") " Nov 03 14:15:31.574646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.574588 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869d7b42-442f-4de3-bcdd-afb8b29a51ce-kube-api-access-qmr2x\" (UniqueName: \"kubernetes.io/projected/869d7b42-442f-4de3-bcdd-afb8b29a51ce-kube-api-access-qmr2x\") pod \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\" (UID: \"869d7b42-442f-4de3-bcdd-afb8b29a51ce\") " Nov 03 14:15:31.575076 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.575053 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-bundle" pod "869d7b42-442f-4de3-bcdd-afb8b29a51ce" (UID: "869d7b42-442f-4de3-bcdd-afb8b29a51ce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:31.577008 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.576982 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869d7b42-442f-4de3-bcdd-afb8b29a51ce-kube-api-access-qmr2x" pod "869d7b42-442f-4de3-bcdd-afb8b29a51ce" (UID: "869d7b42-442f-4de3-bcdd-afb8b29a51ce"). InnerVolumeSpecName "kube-api-access-qmr2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:31.579081 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.579048 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-util" pod "869d7b42-442f-4de3-bcdd-afb8b29a51ce" (UID: "869d7b42-442f-4de3-bcdd-afb8b29a51ce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:31.675704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.675671 6389 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-bundle\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:15:31.675704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.675699 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmr2x\" (UniqueName: \"kubernetes.io/projected/869d7b42-442f-4de3-bcdd-afb8b29a51ce-kube-api-access-qmr2x\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:15:31.675704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:31.675709 6389 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/869d7b42-442f-4de3-bcdd-afb8b29a51ce-util\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:15:32.445906 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:32.445875 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" Nov 03 14:15:32.446390 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:32.445870 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cq5jrn" event={"ID":"869d7b42-442f-4de3-bcdd-afb8b29a51ce","Type":"ContainerDied","Data":"ba0ab28962bcfcd4509376ab5f00c3f554a4dcad7c57bb2be068c0e0c050ba6c"} Nov 03 14:15:32.446390 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:15:32.445997 6389 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba0ab28962bcfcd4509376ab5f00c3f554a4dcad7c57bb2be068c0e0c050ba6c" Nov 03 14:15:58.371559 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:15:58.371515 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:16:09.383246 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383214 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2"] Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383528 6389 cpu_manager.go:414] "containerMap: removing container" podUID="e3c6ef7f-5968-4f97-994c-964c2ea28b4a" containerName="util" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383543 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c6ef7f-5968-4f97-994c-964c2ea28b4a" containerName="util" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383558 6389 cpu_manager.go:414] "containerMap: removing container" podUID="e3c6ef7f-5968-4f97-994c-964c2ea28b4a" containerName="extract" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383563 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c6ef7f-5968-4f97-994c-964c2ea28b4a" containerName="extract" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383569 6389 cpu_manager.go:414] "containerMap: removing container" podUID="869d7b42-442f-4de3-bcdd-afb8b29a51ce" containerName="pull" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383575 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869d7b42-442f-4de3-bcdd-afb8b29a51ce" containerName="pull" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383599 6389 cpu_manager.go:414] "containerMap: removing container" podUID="869d7b42-442f-4de3-bcdd-afb8b29a51ce" containerName="extract" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383619 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869d7b42-442f-4de3-bcdd-afb8b29a51ce" containerName="extract" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383633 6389 cpu_manager.go:414] "containerMap: removing container" podUID="869d7b42-442f-4de3-bcdd-afb8b29a51ce" containerName="util" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383638 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869d7b42-442f-4de3-bcdd-afb8b29a51ce" containerName="util" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383648 6389 cpu_manager.go:414] "containerMap: removing container" podUID="e3c6ef7f-5968-4f97-994c-964c2ea28b4a" containerName="pull" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383652 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c6ef7f-5968-4f97-994c-964c2ea28b4a" containerName="pull" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383709 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="869d7b42-442f-4de3-bcdd-afb8b29a51ce" containerName="extract" Nov 03 14:16:09.383784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.383717 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c6ef7f-5968-4f97-994c-964c2ea28b4a" containerName="extract" Nov 03 14:16:09.385560 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.385543 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" Nov 03 14:16:09.401235 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.401209 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 14:16:09.401235 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.401209 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 14:16:09.411432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.411409 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4qrks\"" Nov 03 14:16:09.411432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.411424 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 14:16:09.412440 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.412419 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 14:16:09.413149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.413127 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2"] Nov 03 14:16:09.462157 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.462123 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsfb8\" (UniqueName: \"kubernetes.io/projected/bf69a67f-797a-4489-a929-789d47efc69c-kube-api-access-vsfb8\") pod \"kuadrant-console-plugin-66c4fb575d-k9lp2\" (UID: \"bf69a67f-797a-4489-a929-789d47efc69c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" Nov 03 14:16:09.462157 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.462160 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf69a67f-797a-4489-a929-789d47efc69c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-k9lp2\" (UID: \"bf69a67f-797a-4489-a929-789d47efc69c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" Nov 03 14:16:09.462387 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.462184 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf69a67f-797a-4489-a929-789d47efc69c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-k9lp2\" (UID: \"bf69a67f-797a-4489-a929-789d47efc69c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" Nov 03 14:16:09.562872 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.562836 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsfb8\" (UniqueName: \"kubernetes.io/projected/bf69a67f-797a-4489-a929-789d47efc69c-kube-api-access-vsfb8\") pod \"kuadrant-console-plugin-66c4fb575d-k9lp2\" (UID: \"bf69a67f-797a-4489-a929-789d47efc69c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" Nov 03 14:16:09.562872 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.562877 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf69a67f-797a-4489-a929-789d47efc69c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-k9lp2\" (UID: \"bf69a67f-797a-4489-a929-789d47efc69c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" Nov 03 14:16:09.563144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.562898 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf69a67f-797a-4489-a929-789d47efc69c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-k9lp2\" (UID: \"bf69a67f-797a-4489-a929-789d47efc69c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" Nov 03 14:16:09.563549 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.563526 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf69a67f-797a-4489-a929-789d47efc69c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-k9lp2\" (UID: \"bf69a67f-797a-4489-a929-789d47efc69c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" Nov 03 14:16:09.565426 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.565406 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf69a67f-797a-4489-a929-789d47efc69c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-k9lp2\" (UID: \"bf69a67f-797a-4489-a929-789d47efc69c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" Nov 03 14:16:09.579016 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.578990 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw"] Nov 03 14:16:09.581127 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.581112 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.584048 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.584013 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsfb8\" (UniqueName: \"kubernetes.io/projected/bf69a67f-797a-4489-a929-789d47efc69c-kube-api-access-vsfb8\") pod \"kuadrant-console-plugin-66c4fb575d-k9lp2\" (UID: \"bf69a67f-797a-4489-a929-789d47efc69c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" Nov 03 14:16:09.605867 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.605843 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 14:16:09.605867 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.605857 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 14:16:09.605867 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.605835 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 14:16:09.605867 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.605853 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 14:16:09.606109 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.605891 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 14:16:09.606109 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.605945 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-m8l2v\"" Nov 03 14:16:09.618861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.618814 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw"] Nov 03 14:16:09.663690 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.663571 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf355776-1f52-4c27-a0fb-8cbbf930cb55-webhook-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.663690 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.663629 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf355776-1f52-4c27-a0fb-8cbbf930cb55-apiservice-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.663915 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.663723 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf355776-1f52-4c27-a0fb-8cbbf930cb55-tmp\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.663915 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.663754 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfdnl\" (UniqueName: \"kubernetes.io/projected/bf355776-1f52-4c27-a0fb-8cbbf930cb55-kube-api-access-gfdnl\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.663915 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.663782 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf355776-1f52-4c27-a0fb-8cbbf930cb55-metrics-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.663915 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.663798 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf355776-1f52-4c27-a0fb-8cbbf930cb55-manifests\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.709498 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.709458 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" Nov 03 14:16:09.764706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.764665 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdnl\" (UniqueName: \"kubernetes.io/projected/bf355776-1f52-4c27-a0fb-8cbbf930cb55-kube-api-access-gfdnl\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.764854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.764710 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf355776-1f52-4c27-a0fb-8cbbf930cb55-metrics-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.764854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.764776 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf355776-1f52-4c27-a0fb-8cbbf930cb55-manifests\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.764974 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.764849 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf355776-1f52-4c27-a0fb-8cbbf930cb55-webhook-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.765703 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.765277 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf355776-1f52-4c27-a0fb-8cbbf930cb55-manifests\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.765703 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.765354 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf355776-1f52-4c27-a0fb-8cbbf930cb55-apiservice-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.765703 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.765426 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf355776-1f52-4c27-a0fb-8cbbf930cb55-tmp\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.768397 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.768255 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf355776-1f52-4c27-a0fb-8cbbf930cb55-metrics-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.768397 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.768366 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf355776-1f52-4c27-a0fb-8cbbf930cb55-webhook-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.768579 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.768442 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf355776-1f52-4c27-a0fb-8cbbf930cb55-apiservice-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.770221 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.770174 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf355776-1f52-4c27-a0fb-8cbbf930cb55-tmp\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.789426 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.789390 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfdnl\" (UniqueName: \"kubernetes.io/projected/bf355776-1f52-4c27-a0fb-8cbbf930cb55-kube-api-access-gfdnl\") pod \"opendatahub-operator-controller-manager-867995fc99-8txkw\" (UID: \"bf355776-1f52-4c27-a0fb-8cbbf930cb55\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:09.898599 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:09.898559 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:10.010156 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:10.010118 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2"] Nov 03 14:16:10.014031 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:16:10.014005 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf69a67f_797a_4489_a929_789d47efc69c.slice/crio-c611b2867c4ddff75e9d457f50ac42110a13795097b5498dcbe83e96ceaa83e6 WatchSource:0}: Error finding container c611b2867c4ddff75e9d457f50ac42110a13795097b5498dcbe83e96ceaa83e6: Status 404 returned error can't find the container with id c611b2867c4ddff75e9d457f50ac42110a13795097b5498dcbe83e96ceaa83e6 Nov 03 14:16:10.246325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:10.246231 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw"] Nov 03 14:16:10.249997 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:16:10.249967 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf355776_1f52_4c27_a0fb_8cbbf930cb55.slice/crio-d5a05f72916336970126042678cc36e13c8ee105eca023dc2170f7161e75360a WatchSource:0}: Error finding container d5a05f72916336970126042678cc36e13c8ee105eca023dc2170f7161e75360a: Status 404 returned error can't find the container with id d5a05f72916336970126042678cc36e13c8ee105eca023dc2170f7161e75360a Nov 03 14:16:10.577082 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:10.577001 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" event={"ID":"bf355776-1f52-4c27-a0fb-8cbbf930cb55","Type":"ContainerStarted","Data":"d5a05f72916336970126042678cc36e13c8ee105eca023dc2170f7161e75360a"} Nov 03 14:16:10.578160 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:10.578140 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" event={"ID":"bf69a67f-797a-4489-a929-789d47efc69c","Type":"ContainerStarted","Data":"c611b2867c4ddff75e9d457f50ac42110a13795097b5498dcbe83e96ceaa83e6"} Nov 03 14:16:14.598799 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:14.598724 6389 generic.go:356] "Generic (PLEG): container finished" podID="bf355776-1f52-4c27-a0fb-8cbbf930cb55" containerID="82733995d09d56635319315449720627557e973a6d3a69add4d3fa9416f05689" exitCode=0 Nov 03 14:16:14.599227 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:14.598814 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" event={"ID":"bf355776-1f52-4c27-a0fb-8cbbf930cb55","Type":"ContainerDied","Data":"82733995d09d56635319315449720627557e973a6d3a69add4d3fa9416f05689"} Nov 03 14:16:15.604223 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:15.604115 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" event={"ID":"bf69a67f-797a-4489-a929-789d47efc69c","Type":"ContainerStarted","Data":"7757908f3ac1be780b26d504827f38e053e464be527f32f4902fffc0455f185f"} Nov 03 14:16:15.605998 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:15.605977 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" event={"ID":"bf355776-1f52-4c27-a0fb-8cbbf930cb55","Type":"ContainerStarted","Data":"9aeb32cef6d4835d92f589165c8e55811eaa2787456d84f7abb839af9da505f7"} Nov 03 14:16:15.606139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:15.606124 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:15.646087 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:15.646040 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k9lp2" podStartSLOduration=1.3534238969999999 podStartE2EDuration="6.646027614s" totalImagesPullingTime="5.292603717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:15.645522066 +0000 UTC m=+251.204158550" watchObservedRunningTime="2026-11-03 14:16:15.646027614 +0000 UTC m=+251.204664097" Nov 03 14:16:15.729521 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:15.729475 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" podStartSLOduration=2.413483439 podStartE2EDuration="6.729460211s" totalImagesPullingTime="3.700340885s" totalInitContainerRuntime="615.635887ms" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:15.718228981 +0000 UTC m=+251.276865476" watchObservedRunningTime="2026-11-03 14:16:15.729460211 +0000 UTC m=+251.288096695" Nov 03 14:16:26.615106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:26.615076 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-8txkw" Nov 03 14:16:28.371994 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:16:28.371945 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:16:38.532027 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.531993 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9"] Nov 03 14:16:38.561531 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.561495 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" Nov 03 14:16:38.564349 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.564321 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9"] Nov 03 14:16:38.599248 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.599221 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502f2df2-d66f-4023-acb6-6ef16ac7f5a8-tmp\") pod \"kube-auth-proxy-76667dfcdc-rj9l9\" (UID: \"502f2df2-d66f-4023-acb6-6ef16ac7f5a8\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" Nov 03 14:16:38.599248 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.599251 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/502f2df2-d66f-4023-acb6-6ef16ac7f5a8-tls-certs\") pod \"kube-auth-proxy-76667dfcdc-rj9l9\" (UID: \"502f2df2-d66f-4023-acb6-6ef16ac7f5a8\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" Nov 03 14:16:38.599430 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.599342 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dkl\" (UniqueName: \"kubernetes.io/projected/502f2df2-d66f-4023-acb6-6ef16ac7f5a8-kube-api-access-m8dkl\") pod \"kube-auth-proxy-76667dfcdc-rj9l9\" (UID: \"502f2df2-d66f-4023-acb6-6ef16ac7f5a8\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" Nov 03 14:16:38.605117 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.605101 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 14:16:38.605256 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.605244 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 14:16:38.605295 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.605261 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-x4mhn\"" Nov 03 14:16:38.700634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.700577 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502f2df2-d66f-4023-acb6-6ef16ac7f5a8-tmp\") pod \"kube-auth-proxy-76667dfcdc-rj9l9\" (UID: \"502f2df2-d66f-4023-acb6-6ef16ac7f5a8\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" Nov 03 14:16:38.700829 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.700647 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/502f2df2-d66f-4023-acb6-6ef16ac7f5a8-tls-certs\") pod \"kube-auth-proxy-76667dfcdc-rj9l9\" (UID: \"502f2df2-d66f-4023-acb6-6ef16ac7f5a8\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" Nov 03 14:16:38.700829 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.700732 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dkl\" (UniqueName: \"kubernetes.io/projected/502f2df2-d66f-4023-acb6-6ef16ac7f5a8-kube-api-access-m8dkl\") pod \"kube-auth-proxy-76667dfcdc-rj9l9\" (UID: \"502f2df2-d66f-4023-acb6-6ef16ac7f5a8\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" Nov 03 14:16:38.702995 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.702971 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/502f2df2-d66f-4023-acb6-6ef16ac7f5a8-tmp\") pod \"kube-auth-proxy-76667dfcdc-rj9l9\" (UID: \"502f2df2-d66f-4023-acb6-6ef16ac7f5a8\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" Nov 03 14:16:38.703241 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.703225 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/502f2df2-d66f-4023-acb6-6ef16ac7f5a8-tls-certs\") pod \"kube-auth-proxy-76667dfcdc-rj9l9\" (UID: \"502f2df2-d66f-4023-acb6-6ef16ac7f5a8\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" Nov 03 14:16:38.720999 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.720968 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dkl\" (UniqueName: \"kubernetes.io/projected/502f2df2-d66f-4023-acb6-6ef16ac7f5a8-kube-api-access-m8dkl\") pod \"kube-auth-proxy-76667dfcdc-rj9l9\" (UID: \"502f2df2-d66f-4023-acb6-6ef16ac7f5a8\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" Nov 03 14:16:38.874106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:38.874016 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" Nov 03 14:16:39.176725 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:39.176696 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9"] Nov 03 14:16:39.179547 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:16:39.179519 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502f2df2_d66f_4023_acb6_6ef16ac7f5a8.slice/crio-b136f2310579a7edbef76fdca4c1444c25e6813d2ca7dc3c210cfb3ae8d072de WatchSource:0}: Error finding container b136f2310579a7edbef76fdca4c1444c25e6813d2ca7dc3c210cfb3ae8d072de: Status 404 returned error can't find the container with id b136f2310579a7edbef76fdca4c1444c25e6813d2ca7dc3c210cfb3ae8d072de Nov 03 14:16:39.699750 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:39.699712 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" event={"ID":"502f2df2-d66f-4023-acb6-6ef16ac7f5a8","Type":"ContainerStarted","Data":"b136f2310579a7edbef76fdca4c1444c25e6813d2ca7dc3c210cfb3ae8d072de"} Nov 03 14:16:41.710665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:41.710628 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" event={"ID":"502f2df2-d66f-4023-acb6-6ef16ac7f5a8","Type":"ContainerStarted","Data":"a49b0acc20c3b0ab1d6bd5a3898f05a5aa43041dd0e6b084cb2bb471eb6ce53c"} Nov 03 14:16:41.787112 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:41.787056 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-76667dfcdc-rj9l9" podStartSLOduration=1.367985284 podStartE2EDuration="3.787042489s" totalImagesPullingTime="2.419057205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:41.786275361 +0000 UTC m=+277.344911845" watchObservedRunningTime="2026-11-03 14:16:41.787042489 +0000 UTC m=+277.345678966" Nov 03 14:16:42.869624 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:42.869576 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj"] Nov 03 14:16:42.873116 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:42.873053 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:42.888641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:42.888599 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-jj6cp\"" Nov 03 14:16:42.899994 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:42.899969 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj"] Nov 03 14:16:42.943965 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:42.943938 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5skzm\" (UniqueName: \"kubernetes.io/projected/f884e514-a7a4-459f-aba9-62f7ce76671a-kube-api-access-5skzm\") pod \"kserve-module-controller-manager-7566cf5fc8-qdflj\" (UID: \"f884e514-a7a4-459f-aba9-62f7ce76671a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:42.944112 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:42.943969 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f884e514-a7a4-459f-aba9-62f7ce76671a-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-qdflj\" (UID: \"f884e514-a7a4-459f-aba9-62f7ce76671a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:42.944112 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:42.944011 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f884e514-a7a4-459f-aba9-62f7ce76671a-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-qdflj\" (UID: \"f884e514-a7a4-459f-aba9-62f7ce76671a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:42.944214 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:42.944190 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f884e514-a7a4-459f-aba9-62f7ce76671a-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-qdflj\" (UID: \"f884e514-a7a4-459f-aba9-62f7ce76671a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:42.962879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:42.962840 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 14:16:42.963024 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:42.962882 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 14:16:42.986084 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:42.986062 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 14:16:43.045208 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:43.045181 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f884e514-a7a4-459f-aba9-62f7ce76671a-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-qdflj\" (UID: \"f884e514-a7a4-459f-aba9-62f7ce76671a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:43.045294 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:43.045236 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f884e514-a7a4-459f-aba9-62f7ce76671a-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-qdflj\" (UID: \"f884e514-a7a4-459f-aba9-62f7ce76671a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:43.045294 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:43.045264 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5skzm\" (UniqueName: \"kubernetes.io/projected/f884e514-a7a4-459f-aba9-62f7ce76671a-kube-api-access-5skzm\") pod \"kserve-module-controller-manager-7566cf5fc8-qdflj\" (UID: \"f884e514-a7a4-459f-aba9-62f7ce76671a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:43.045294 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:43.045288 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f884e514-a7a4-459f-aba9-62f7ce76671a-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-qdflj\" (UID: \"f884e514-a7a4-459f-aba9-62f7ce76671a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:43.045633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:43.045589 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f884e514-a7a4-459f-aba9-62f7ce76671a-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-qdflj\" (UID: \"f884e514-a7a4-459f-aba9-62f7ce76671a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:43.045723 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:43.045667 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f884e514-a7a4-459f-aba9-62f7ce76671a-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-qdflj\" (UID: \"f884e514-a7a4-459f-aba9-62f7ce76671a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:43.046166 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:43.046144 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f884e514-a7a4-459f-aba9-62f7ce76671a-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-qdflj\" (UID: \"f884e514-a7a4-459f-aba9-62f7ce76671a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:43.065754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:43.065737 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5skzm\" (UniqueName: \"kubernetes.io/projected/f884e514-a7a4-459f-aba9-62f7ce76671a-kube-api-access-5skzm\") pod \"kserve-module-controller-manager-7566cf5fc8-qdflj\" (UID: \"f884e514-a7a4-459f-aba9-62f7ce76671a\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:43.185411 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:43.185379 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:43.497656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:43.497631 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj"] Nov 03 14:16:43.500212 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:16:43.500184 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf884e514_a7a4_459f_aba9_62f7ce76671a.slice/crio-39385bac3128225d27fa4f5bef42c6d31ebea253f8776e0ec5cb9a1452fc2229 WatchSource:0}: Error finding container 39385bac3128225d27fa4f5bef42c6d31ebea253f8776e0ec5cb9a1452fc2229: Status 404 returned error can't find the container with id 39385bac3128225d27fa4f5bef42c6d31ebea253f8776e0ec5cb9a1452fc2229 Nov 03 14:16:43.719753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:43.719715 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" event={"ID":"f884e514-a7a4-459f-aba9-62f7ce76671a","Type":"ContainerStarted","Data":"39385bac3128225d27fa4f5bef42c6d31ebea253f8776e0ec5cb9a1452fc2229"} Nov 03 14:16:45.729753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:45.729719 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" event={"ID":"f884e514-a7a4-459f-aba9-62f7ce76671a","Type":"ContainerStarted","Data":"4e4eaa91d2ba8e0a53bc5ff66bf0ca6c87db71f5b2a19e26877927fdc2cde606"} Nov 03 14:16:45.730124 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:45.729859 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:16:45.784182 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:45.784133 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" podStartSLOduration=2.038890259 podStartE2EDuration="3.784114929s" totalImagesPullingTime="1.74522467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:45.783650241 +0000 UTC m=+281.342286721" watchObservedRunningTime="2026-11-03 14:16:45.784114929 +0000 UTC m=+281.342751414" Nov 03 14:16:48.356999 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.356934 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ff8bx"] Nov 03 14:16:48.360447 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.360426 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" Nov 03 14:16:48.379406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.379384 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 14:16:48.381711 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.381688 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ff8bx"] Nov 03 14:16:48.389228 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.389200 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-config-file\") pod \"limitador-limitador-74dfcfb894-ff8bx\" (UID: \"bb395ecc-be36-4b87-a9d8-d3cbb464b63c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" Nov 03 14:16:48.389362 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.389237 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrstv\" (UniqueName: \"kubernetes.io/projected/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-kube-api-access-rrstv\") pod \"limitador-limitador-74dfcfb894-ff8bx\" (UID: \"bb395ecc-be36-4b87-a9d8-d3cbb464b63c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" Nov 03 14:16:48.490112 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.490080 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-config-file\") pod \"limitador-limitador-74dfcfb894-ff8bx\" (UID: \"bb395ecc-be36-4b87-a9d8-d3cbb464b63c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" Nov 03 14:16:48.490297 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.490138 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrstv\" (UniqueName: \"kubernetes.io/projected/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-kube-api-access-rrstv\") pod \"limitador-limitador-74dfcfb894-ff8bx\" (UID: \"bb395ecc-be36-4b87-a9d8-d3cbb464b63c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" Nov 03 14:16:48.490771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.490751 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-config-file\") pod \"limitador-limitador-74dfcfb894-ff8bx\" (UID: \"bb395ecc-be36-4b87-a9d8-d3cbb464b63c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" Nov 03 14:16:48.491257 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.491237 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ff8bx"] Nov 03 14:16:48.511102 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.511072 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrstv\" (UniqueName: \"kubernetes.io/projected/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-kube-api-access-rrstv\") pod \"limitador-limitador-74dfcfb894-ff8bx\" (UID: \"bb395ecc-be36-4b87-a9d8-d3cbb464b63c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" Nov 03 14:16:48.673519 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.673493 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" Nov 03 14:16:48.969753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:48.969624 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ff8bx"] Nov 03 14:16:48.972283 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:16:48.972256 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb395ecc_be36_4b87_a9d8_d3cbb464b63c.slice/crio-96930eb2c2f8c0a2fe0fe449a82e5b5637cc1383bcb65bd573edb11dc479c382 WatchSource:0}: Error finding container 96930eb2c2f8c0a2fe0fe449a82e5b5637cc1383bcb65bd573edb11dc479c382: Status 404 returned error can't find the container with id 96930eb2c2f8c0a2fe0fe449a82e5b5637cc1383bcb65bd573edb11dc479c382 Nov 03 14:16:49.750545 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:49.750471 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" event={"ID":"bb395ecc-be36-4b87-a9d8-d3cbb464b63c","Type":"ContainerStarted","Data":"96930eb2c2f8c0a2fe0fe449a82e5b5637cc1383bcb65bd573edb11dc479c382"} Nov 03 14:16:51.759950 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:51.759913 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" event={"ID":"bb395ecc-be36-4b87-a9d8-d3cbb464b63c","Type":"ContainerStarted","Data":"5259fdd3afb7f2c6863b2f63dfd4ec20e3c2d60d91c4362a3a993d91d12a466d"} Nov 03 14:16:51.760352 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:51.760055 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" Nov 03 14:16:51.800199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:16:51.800151 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" podStartSLOduration=1.509473282 podStartE2EDuration="3.800139115s" totalImagesPullingTime="2.290665833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:51.799573224 +0000 UTC m=+287.358209707" watchObservedRunningTime="2026-11-03 14:16:51.800139115 +0000 UTC m=+287.358775599" Nov 03 14:16:58.372665 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:16:58.372623 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:17:02.765234 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:02.765207 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" Nov 03 14:17:05.360769 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:05.360736 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ff8bx"] Nov 03 14:17:05.361255 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:05.361078 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" podUID="bb395ecc-be36-4b87-a9d8-d3cbb464b63c" containerName="limitador" containerID="cri-o://5259fdd3afb7f2c6863b2f63dfd4ec20e3c2d60d91c4362a3a993d91d12a466d" gracePeriod=30 Nov 03 14:17:05.814442 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:05.814410 6389 generic.go:356] "Generic (PLEG): container finished" podID="bb395ecc-be36-4b87-a9d8-d3cbb464b63c" containerID="5259fdd3afb7f2c6863b2f63dfd4ec20e3c2d60d91c4362a3a993d91d12a466d" exitCode=0 Nov 03 14:17:05.814637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:05.814489 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" event={"ID":"bb395ecc-be36-4b87-a9d8-d3cbb464b63c","Type":"ContainerDied","Data":"5259fdd3afb7f2c6863b2f63dfd4ec20e3c2d60d91c4362a3a993d91d12a466d"} Nov 03 14:17:06.295134 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.295109 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" Nov 03 14:17:06.446673 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.446568 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-config-file\" (UniqueName: \"kubernetes.io/configmap/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-config-file\") pod \"bb395ecc-be36-4b87-a9d8-d3cbb464b63c\" (UID: \"bb395ecc-be36-4b87-a9d8-d3cbb464b63c\") " Nov 03 14:17:06.446673 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.446649 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-kube-api-access-rrstv\" (UniqueName: \"kubernetes.io/projected/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-kube-api-access-rrstv\") pod \"bb395ecc-be36-4b87-a9d8-d3cbb464b63c\" (UID: \"bb395ecc-be36-4b87-a9d8-d3cbb464b63c\") " Nov 03 14:17:06.447168 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.446901 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-config-file" pod "bb395ecc-be36-4b87-a9d8-d3cbb464b63c" (UID: "bb395ecc-be36-4b87-a9d8-d3cbb464b63c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:17:06.449528 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.449498 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-kube-api-access-rrstv" pod "bb395ecc-be36-4b87-a9d8-d3cbb464b63c" (UID: "bb395ecc-be36-4b87-a9d8-d3cbb464b63c"). InnerVolumeSpecName "kube-api-access-rrstv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:17:06.548205 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.548153 6389 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-config-file\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:17:06.548205 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.548201 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrstv\" (UniqueName: \"kubernetes.io/projected/bb395ecc-be36-4b87-a9d8-d3cbb464b63c-kube-api-access-rrstv\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:17:06.734880 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.734801 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qdflj" Nov 03 14:17:06.819693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.819657 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" event={"ID":"bb395ecc-be36-4b87-a9d8-d3cbb464b63c","Type":"ContainerDied","Data":"96930eb2c2f8c0a2fe0fe449a82e5b5637cc1383bcb65bd573edb11dc479c382"} Nov 03 14:17:06.819693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.819694 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ff8bx" Nov 03 14:17:06.819905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.819726 6389 scope.go:122] "RemoveContainer" containerID="5259fdd3afb7f2c6863b2f63dfd4ec20e3c2d60d91c4362a3a993d91d12a466d" Nov 03 14:17:06.870780 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.870751 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ff8bx"] Nov 03 14:17:06.875863 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:06.875838 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ff8bx"] Nov 03 14:17:08.616460 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:08.616428 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb395ecc-be36-4b87-a9d8-d3cbb464b63c" path="/var/lib/kubelet/pods/bb395ecc-be36-4b87-a9d8-d3cbb464b63c/volumes" Nov 03 14:17:11.299378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.299346 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd"] Nov 03 14:17:11.299826 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.299706 6389 cpu_manager.go:414] "containerMap: removing container" podUID="bb395ecc-be36-4b87-a9d8-d3cbb464b63c" containerName="limitador" Nov 03 14:17:11.299826 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.299720 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb395ecc-be36-4b87-a9d8-d3cbb464b63c" containerName="limitador" Nov 03 14:17:11.299935 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.299835 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb395ecc-be36-4b87-a9d8-d3cbb464b63c" containerName="limitador" Nov 03 14:17:11.304021 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.304002 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" Nov 03 14:17:11.321197 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.321175 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 14:17:11.321337 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.321204 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd"] Nov 03 14:17:11.321460 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.321443 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:17:11.331500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.331465 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:17:11.331500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.331471 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fmjht\"" Nov 03 14:17:11.388421 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.388394 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5e2b48a3-4d88-4851-86f3-cd8bcc0292c0-data\") pod \"postgres-7cdbf6dc44-qpxtd\" (UID: \"5e2b48a3-4d88-4851-86f3-cd8bcc0292c0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" Nov 03 14:17:11.388421 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.388432 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmsk7\" (UniqueName: \"kubernetes.io/projected/5e2b48a3-4d88-4851-86f3-cd8bcc0292c0-kube-api-access-hmsk7\") pod \"postgres-7cdbf6dc44-qpxtd\" (UID: \"5e2b48a3-4d88-4851-86f3-cd8bcc0292c0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" Nov 03 14:17:11.489243 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.489206 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5e2b48a3-4d88-4851-86f3-cd8bcc0292c0-data\") pod \"postgres-7cdbf6dc44-qpxtd\" (UID: \"5e2b48a3-4d88-4851-86f3-cd8bcc0292c0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" Nov 03 14:17:11.489410 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.489249 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsk7\" (UniqueName: \"kubernetes.io/projected/5e2b48a3-4d88-4851-86f3-cd8bcc0292c0-kube-api-access-hmsk7\") pod \"postgres-7cdbf6dc44-qpxtd\" (UID: \"5e2b48a3-4d88-4851-86f3-cd8bcc0292c0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" Nov 03 14:17:11.489600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.489582 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5e2b48a3-4d88-4851-86f3-cd8bcc0292c0-data\") pod \"postgres-7cdbf6dc44-qpxtd\" (UID: \"5e2b48a3-4d88-4851-86f3-cd8bcc0292c0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" Nov 03 14:17:11.510139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.510100 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmsk7\" (UniqueName: \"kubernetes.io/projected/5e2b48a3-4d88-4851-86f3-cd8bcc0292c0-kube-api-access-hmsk7\") pod \"postgres-7cdbf6dc44-qpxtd\" (UID: \"5e2b48a3-4d88-4851-86f3-cd8bcc0292c0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" Nov 03 14:17:11.616759 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.616671 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" Nov 03 14:17:11.923901 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:11.923862 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd"] Nov 03 14:17:11.926828 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:17:11.926803 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2b48a3_4d88_4851_86f3_cd8bcc0292c0.slice/crio-7b6e0989e4477aa4a2d078dcee4fd12f23389cb2b0113b115b5832407cbeafb9 WatchSource:0}: Error finding container 7b6e0989e4477aa4a2d078dcee4fd12f23389cb2b0113b115b5832407cbeafb9: Status 404 returned error can't find the container with id 7b6e0989e4477aa4a2d078dcee4fd12f23389cb2b0113b115b5832407cbeafb9 Nov 03 14:17:12.855934 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:12.855811 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" event={"ID":"5e2b48a3-4d88-4851-86f3-cd8bcc0292c0","Type":"ContainerStarted","Data":"7b6e0989e4477aa4a2d078dcee4fd12f23389cb2b0113b115b5832407cbeafb9"} Nov 03 14:17:17.879159 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:17.879117 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" event={"ID":"5e2b48a3-4d88-4851-86f3-cd8bcc0292c0","Type":"ContainerStarted","Data":"f2ff4e3f7da7d4f21cdaf7fb8009763d5d0356795f09b1a5af6bab629e09c7a5"} Nov 03 14:17:17.879600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:17.879252 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" Nov 03 14:17:17.919049 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:17.919006 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" podStartSLOduration=1.780718947 podStartE2EDuration="6.918992373s" totalImagesPullingTime="5.138273426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:17:17.918552765 +0000 UTC m=+313.477189249" watchObservedRunningTime="2026-11-03 14:17:17.918992373 +0000 UTC m=+313.477628856" Nov 03 14:17:23.912382 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:23.912354 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qpxtd" Nov 03 14:17:28.373968 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:17:28.373924 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:17:31.100379 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.100345 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64678f5d5d-96mcs"] Nov 03 14:17:31.107137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.107113 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:31.125000 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.124967 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-62kg8\"" Nov 03 14:17:31.126956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.126930 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 14:17:31.136362 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.136336 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 14:17:31.136529 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.136451 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 14:17:31.137947 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.137925 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64678f5d5d-96mcs"] Nov 03 14:17:31.139080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.139058 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c646z\" (UniqueName: \"kubernetes.io/projected/1859e393-7fa6-4046-b2df-132df6e29949-kube-api-access-c646z\") pod \"maas-controller-64678f5d5d-96mcs\" (UID: \"1859e393-7fa6-4046-b2df-132df6e29949\") " pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:31.139144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.139099 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-webhook-cert\") pod \"maas-controller-64678f5d5d-96mcs\" (UID: \"1859e393-7fa6-4046-b2df-132df6e29949\") " pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:31.139266 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.139243 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-metrics-certs\") pod \"maas-controller-64678f5d5d-96mcs\" (UID: \"1859e393-7fa6-4046-b2df-132df6e29949\") " pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:31.239896 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.239861 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c646z\" (UniqueName: \"kubernetes.io/projected/1859e393-7fa6-4046-b2df-132df6e29949-kube-api-access-c646z\") pod \"maas-controller-64678f5d5d-96mcs\" (UID: \"1859e393-7fa6-4046-b2df-132df6e29949\") " pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:31.240097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.239907 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-webhook-cert\") pod \"maas-controller-64678f5d5d-96mcs\" (UID: \"1859e393-7fa6-4046-b2df-132df6e29949\") " pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:31.240097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.239981 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-metrics-certs\") pod \"maas-controller-64678f5d5d-96mcs\" (UID: \"1859e393-7fa6-4046-b2df-132df6e29949\") " pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:31.242603 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.242579 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-metrics-certs\") pod \"maas-controller-64678f5d5d-96mcs\" (UID: \"1859e393-7fa6-4046-b2df-132df6e29949\") " pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:31.242720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.242683 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-webhook-cert\") pod \"maas-controller-64678f5d5d-96mcs\" (UID: \"1859e393-7fa6-4046-b2df-132df6e29949\") " pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:31.263894 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.263871 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c646z\" (UniqueName: \"kubernetes.io/projected/1859e393-7fa6-4046-b2df-132df6e29949-kube-api-access-c646z\") pod \"maas-controller-64678f5d5d-96mcs\" (UID: \"1859e393-7fa6-4046-b2df-132df6e29949\") " pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:31.421053 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.421018 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:31.609524 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.609488 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-jcn8k"] Nov 03 14:17:31.614247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.614215 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:31.642567 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.641926 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-jcn8k"] Nov 03 14:17:31.643681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.643654 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdvk\" (UniqueName: \"kubernetes.io/projected/93952091-e0db-4779-9d1a-fab69c52f73f-kube-api-access-7wdvk\") pod \"maas-controller-6dfdc587c5-jcn8k\" (UID: \"93952091-e0db-4779-9d1a-fab69c52f73f\") " pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:31.643846 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.643735 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-metrics-certs\") pod \"maas-controller-6dfdc587c5-jcn8k\" (UID: \"93952091-e0db-4779-9d1a-fab69c52f73f\") " pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:31.643846 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.643798 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-webhook-cert\") pod \"maas-controller-6dfdc587c5-jcn8k\" (UID: \"93952091-e0db-4779-9d1a-fab69c52f73f\") " pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:31.721154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.721068 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64678f5d5d-96mcs"] Nov 03 14:17:31.725112 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:17:31.725081 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1859e393_7fa6_4046_b2df_132df6e29949.slice/crio-b2aa6bd3e4f171bfe190627b414c7311028b3425ac25716e9e81fa9abc73b6e4 WatchSource:0}: Error finding container b2aa6bd3e4f171bfe190627b414c7311028b3425ac25716e9e81fa9abc73b6e4: Status 404 returned error can't find the container with id b2aa6bd3e4f171bfe190627b414c7311028b3425ac25716e9e81fa9abc73b6e4 Nov 03 14:17:31.744708 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.744678 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-metrics-certs\") pod \"maas-controller-6dfdc587c5-jcn8k\" (UID: \"93952091-e0db-4779-9d1a-fab69c52f73f\") " pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:31.744849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.744734 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-webhook-cert\") pod \"maas-controller-6dfdc587c5-jcn8k\" (UID: \"93952091-e0db-4779-9d1a-fab69c52f73f\") " pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:31.744849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.744780 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdvk\" (UniqueName: \"kubernetes.io/projected/93952091-e0db-4779-9d1a-fab69c52f73f-kube-api-access-7wdvk\") pod \"maas-controller-6dfdc587c5-jcn8k\" (UID: \"93952091-e0db-4779-9d1a-fab69c52f73f\") " pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:31.747331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.747307 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-webhook-cert\") pod \"maas-controller-6dfdc587c5-jcn8k\" (UID: \"93952091-e0db-4779-9d1a-fab69c52f73f\") " pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:31.747443 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.747352 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-metrics-certs\") pod \"maas-controller-6dfdc587c5-jcn8k\" (UID: \"93952091-e0db-4779-9d1a-fab69c52f73f\") " pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:31.765147 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.765112 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdvk\" (UniqueName: \"kubernetes.io/projected/93952091-e0db-4779-9d1a-fab69c52f73f-kube-api-access-7wdvk\") pod \"maas-controller-6dfdc587c5-jcn8k\" (UID: \"93952091-e0db-4779-9d1a-fab69c52f73f\") " pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:31.934598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.934562 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64678f5d5d-96mcs" event={"ID":"1859e393-7fa6-4046-b2df-132df6e29949","Type":"ContainerStarted","Data":"b2aa6bd3e4f171bfe190627b414c7311028b3425ac25716e9e81fa9abc73b6e4"} Nov 03 14:17:31.942901 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:31.942880 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:32.244338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:32.244273 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-jcn8k"] Nov 03 14:17:32.247173 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:17:32.247137 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93952091_e0db_4779_9d1a_fab69c52f73f.slice/crio-129b9268cc77f245f751ee75dfe1509f29cdb5159afa5ffef95c235e4c75b3b8 WatchSource:0}: Error finding container 129b9268cc77f245f751ee75dfe1509f29cdb5159afa5ffef95c235e4c75b3b8: Status 404 returned error can't find the container with id 129b9268cc77f245f751ee75dfe1509f29cdb5159afa5ffef95c235e4c75b3b8 Nov 03 14:17:32.945048 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:32.944949 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" event={"ID":"93952091-e0db-4779-9d1a-fab69c52f73f","Type":"ContainerStarted","Data":"129b9268cc77f245f751ee75dfe1509f29cdb5159afa5ffef95c235e4c75b3b8"} Nov 03 14:17:34.958597 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:34.958550 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64678f5d5d-96mcs" event={"ID":"1859e393-7fa6-4046-b2df-132df6e29949","Type":"ContainerStarted","Data":"1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9"} Nov 03 14:17:34.958996 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:34.958843 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:35.016252 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:35.016158 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64678f5d5d-96mcs" podStartSLOduration=1.025209361 podStartE2EDuration="4.016141s" totalImagesPullingTime="2.990931639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:17:35.015030392 +0000 UTC m=+330.573666875" watchObservedRunningTime="2026-11-03 14:17:35.016141 +0000 UTC m=+330.574777488" Nov 03 14:17:35.964958 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:35.964921 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" event={"ID":"93952091-e0db-4779-9d1a-fab69c52f73f","Type":"ContainerStarted","Data":"b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8"} Nov 03 14:17:35.965428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:35.965015 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:36.009517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:36.009456 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" podStartSLOduration=2.208919037 podStartE2EDuration="5.009439977s" totalImagesPullingTime="2.80052094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:17:36.00854204 +0000 UTC m=+331.567178524" watchObservedRunningTime="2026-11-03 14:17:36.009439977 +0000 UTC m=+331.568076465" Nov 03 14:17:38.356592 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.356547 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl"] Nov 03 14:17:38.362148 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.362128 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.386515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.386473 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl"] Nov 03 14:17:38.412147 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.412115 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0ab114-50e5-4549-8af4-f361d3f23df7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.412147 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.412148 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q866w\" (UniqueName: \"kubernetes.io/projected/ea0ab114-50e5-4549-8af4-f361d3f23df7-kube-api-access-q866w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.412328 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.412173 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea0ab114-50e5-4549-8af4-f361d3f23df7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.412328 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.412280 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0ab114-50e5-4549-8af4-f361d3f23df7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.412328 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.412324 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ea0ab114-50e5-4549-8af4-f361d3f23df7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.412421 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.412412 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ea0ab114-50e5-4549-8af4-f361d3f23df7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.412455 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.412431 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0ab114-50e5-4549-8af4-f361d3f23df7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.513359 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.513320 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ea0ab114-50e5-4549-8af4-f361d3f23df7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.513359 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.513360 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0ab114-50e5-4549-8af4-f361d3f23df7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.513587 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.513409 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0ab114-50e5-4549-8af4-f361d3f23df7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.513587 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.513431 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q866w\" (UniqueName: \"kubernetes.io/projected/ea0ab114-50e5-4549-8af4-f361d3f23df7-kube-api-access-q866w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.513587 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.513455 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea0ab114-50e5-4549-8af4-f361d3f23df7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.513760 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.513485 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0ab114-50e5-4549-8af4-f361d3f23df7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.513760 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.513686 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ea0ab114-50e5-4549-8af4-f361d3f23df7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.514470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.514445 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0ab114-50e5-4549-8af4-f361d3f23df7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.516081 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.516061 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ea0ab114-50e5-4549-8af4-f361d3f23df7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.516395 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.516376 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea0ab114-50e5-4549-8af4-f361d3f23df7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.516517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.516496 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ea0ab114-50e5-4549-8af4-f361d3f23df7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.516734 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.516717 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0ab114-50e5-4549-8af4-f361d3f23df7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.533927 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.533900 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0ab114-50e5-4549-8af4-f361d3f23df7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.534574 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.534550 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q866w\" (UniqueName: \"kubernetes.io/projected/ea0ab114-50e5-4549-8af4-f361d3f23df7-kube-api-access-q866w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m6wvl\" (UID: \"ea0ab114-50e5-4549-8af4-f361d3f23df7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:38.674517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:38.674481 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:39.009728 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:39.009690 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl"] Nov 03 14:17:39.012671 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:17:39.012643 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0ab114_50e5_4549_8af4_f361d3f23df7.slice/crio-5fb9fac5c08d32d139501f933d1ec8eda239756ea351a3ef1265ea56509f5e21 WatchSource:0}: Error finding container 5fb9fac5c08d32d139501f933d1ec8eda239756ea351a3ef1265ea56509f5e21: Status 404 returned error can't find the container with id 5fb9fac5c08d32d139501f933d1ec8eda239756ea351a3ef1265ea56509f5e21 Nov 03 14:17:39.015037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:39.015001 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:17:39.015123 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:39.015088 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:17:39.982415 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:39.982383 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" event={"ID":"ea0ab114-50e5-4549-8af4-f361d3f23df7","Type":"ContainerStarted","Data":"f2e38c66c1c174bfb9f2933dad369f6f884de8709af93f1d19ecfc843432c289"} Nov 03 14:17:39.982415 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:39.982430 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" event={"ID":"ea0ab114-50e5-4549-8af4-f361d3f23df7","Type":"ContainerStarted","Data":"5fb9fac5c08d32d139501f933d1ec8eda239756ea351a3ef1265ea56509f5e21"} Nov 03 14:17:39.982995 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:39.982517 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:40.031974 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:40.029971 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" podStartSLOduration=2.029950619 podStartE2EDuration="2.029950619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:17:40.028885029 +0000 UTC m=+335.587521514" watchObservedRunningTime="2026-11-03 14:17:40.029950619 +0000 UTC m=+335.588587105" Nov 03 14:17:40.988850 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:40.988820 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m6wvl" Nov 03 14:17:41.091888 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.091854 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk"] Nov 03 14:17:41.092116 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.092086 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" podUID="90f82a42-a4ff-43dc-96c5-56135f30576a" containerName="discovery" containerID="cri-o://458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61" gracePeriod=30 Nov 03 14:17:41.338528 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.338505 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:17:41.437096 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.437060 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-kube-api-access-vhljv\" (UniqueName: \"kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-kube-api-access-vhljv\") pod \"90f82a42-a4ff-43dc-96c5-56135f30576a\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " Nov 03 14:17:41.437288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.437108 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-token\" (UniqueName: \"kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-token\") pod \"90f82a42-a4ff-43dc-96c5-56135f30576a\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " Nov 03 14:17:41.437288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.437159 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-kubeconfig\") pod \"90f82a42-a4ff-43dc-96c5-56135f30576a\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " Nov 03 14:17:41.437288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.437183 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-dns-cert\") pod \"90f82a42-a4ff-43dc-96c5-56135f30576a\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " Nov 03 14:17:41.437288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.437207 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-ca-configmap\") pod \"90f82a42-a4ff-43dc-96c5-56135f30576a\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " Nov 03 14:17:41.437288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.437244 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f82a42-a4ff-43dc-96c5-56135f30576a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f82a42-a4ff-43dc-96c5-56135f30576a-local-certs\") pod \"90f82a42-a4ff-43dc-96c5-56135f30576a\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " Nov 03 14:17:41.437288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.437284 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-cacerts\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-cacerts\") pod \"90f82a42-a4ff-43dc-96c5-56135f30576a\" (UID: \"90f82a42-a4ff-43dc-96c5-56135f30576a\") " Nov 03 14:17:41.437704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.437673 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-ca-configmap" pod "90f82a42-a4ff-43dc-96c5-56135f30576a" (UID: "90f82a42-a4ff-43dc-96c5-56135f30576a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:17:41.439821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.439769 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-token" pod "90f82a42-a4ff-43dc-96c5-56135f30576a" (UID: "90f82a42-a4ff-43dc-96c5-56135f30576a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:17:41.439821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.439808 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-dns-cert" pod "90f82a42-a4ff-43dc-96c5-56135f30576a" (UID: "90f82a42-a4ff-43dc-96c5-56135f30576a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:17:41.440129 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.440101 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f82a42-a4ff-43dc-96c5-56135f30576a-local-certs" pod "90f82a42-a4ff-43dc-96c5-56135f30576a" (UID: "90f82a42-a4ff-43dc-96c5-56135f30576a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:17:41.440227 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.440203 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-cacerts" pod "90f82a42-a4ff-43dc-96c5-56135f30576a" (UID: "90f82a42-a4ff-43dc-96c5-56135f30576a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:17:41.440284 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.440256 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-kube-api-access-vhljv" pod "90f82a42-a4ff-43dc-96c5-56135f30576a" (UID: "90f82a42-a4ff-43dc-96c5-56135f30576a"). InnerVolumeSpecName "kube-api-access-vhljv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:17:41.440319 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.440305 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-kubeconfig" pod "90f82a42-a4ff-43dc-96c5-56135f30576a" (UID: "90f82a42-a4ff-43dc-96c5-56135f30576a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:17:41.538111 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.538032 6389 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-kubeconfig\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:17:41.538111 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.538060 6389 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-dns-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:17:41.538111 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.538073 6389 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-csr-ca-configmap\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:17:41.538111 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.538083 6389 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f82a42-a4ff-43dc-96c5-56135f30576a-local-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:17:41.538111 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.538091 6389 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90f82a42-a4ff-43dc-96c5-56135f30576a-cacerts\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:17:41.538111 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.538099 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhljv\" (UniqueName: \"kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-kube-api-access-vhljv\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:17:41.538111 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.538107 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f82a42-a4ff-43dc-96c5-56135f30576a-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:17:41.991829 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.991796 6389 generic.go:356] "Generic (PLEG): container finished" podID="90f82a42-a4ff-43dc-96c5-56135f30576a" containerID="458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61" exitCode=0 Nov 03 14:17:41.992264 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.991838 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" Nov 03 14:17:41.992264 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.991870 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" event={"ID":"90f82a42-a4ff-43dc-96c5-56135f30576a","Type":"ContainerDied","Data":"458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61"} Nov 03 14:17:41.992264 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.991920 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk" event={"ID":"90f82a42-a4ff-43dc-96c5-56135f30576a","Type":"ContainerDied","Data":"6ea80635f4575e0e98d4c3d63ef0a2fb903c9eeaeb253a71b70f82af85be2782"} Nov 03 14:17:41.992264 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:41.991940 6389 scope.go:122] "RemoveContainer" containerID="458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61" Nov 03 14:17:42.001110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:42.001093 6389 scope.go:122] "RemoveContainer" containerID="458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61" Nov 03 14:17:42.001388 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:17:42.001367 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61\": container with ID starting with 458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61 not found: ID does not exist" containerID="458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61" Nov 03 14:17:42.001456 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:42.001401 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61"} err="failed to get container status \"458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61\": rpc error: code = NotFound desc = could not find container \"458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61\": container with ID starting with 458a457da20ad2e8331ac004de6359c006795f201e892841af14de3303119c61 not found: ID does not exist" Nov 03 14:17:42.042893 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:42.042858 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk"] Nov 03 14:17:42.054168 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:42.054131 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mf4wk"] Nov 03 14:17:42.616544 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:42.616512 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f82a42-a4ff-43dc-96c5-56135f30576a" path="/var/lib/kubelet/pods/90f82a42-a4ff-43dc-96c5-56135f30576a/volumes" Nov 03 14:17:45.966876 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:45.966841 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:46.970838 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:46.970805 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:17:47.054517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:47.054487 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64678f5d5d-96mcs"] Nov 03 14:17:47.054802 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:47.054718 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-64678f5d5d-96mcs" podUID="1859e393-7fa6-4046-b2df-132df6e29949" containerName="manager" containerID="cri-o://1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9" gracePeriod=30 Nov 03 14:17:47.307550 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:47.307524 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:47.391526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:47.391494 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-webhook-cert\" (UniqueName: \"kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-webhook-cert\") pod \"1859e393-7fa6-4046-b2df-132df6e29949\" (UID: \"1859e393-7fa6-4046-b2df-132df6e29949\") " Nov 03 14:17:47.391526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:47.391532 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1859e393-7fa6-4046-b2df-132df6e29949-kube-api-access-c646z\" (UniqueName: \"kubernetes.io/projected/1859e393-7fa6-4046-b2df-132df6e29949-kube-api-access-c646z\") pod \"1859e393-7fa6-4046-b2df-132df6e29949\" (UID: \"1859e393-7fa6-4046-b2df-132df6e29949\") " Nov 03 14:17:47.391865 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:47.391578 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-metrics-certs\") pod \"1859e393-7fa6-4046-b2df-132df6e29949\" (UID: \"1859e393-7fa6-4046-b2df-132df6e29949\") " Nov 03 14:17:47.393930 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:47.393887 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-webhook-cert" pod "1859e393-7fa6-4046-b2df-132df6e29949" (UID: "1859e393-7fa6-4046-b2df-132df6e29949"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:17:47.393930 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:47.393898 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-metrics-certs" pod "1859e393-7fa6-4046-b2df-132df6e29949" (UID: "1859e393-7fa6-4046-b2df-132df6e29949"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:17:47.394121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:47.394098 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1859e393-7fa6-4046-b2df-132df6e29949-kube-api-access-c646z" pod "1859e393-7fa6-4046-b2df-132df6e29949" (UID: "1859e393-7fa6-4046-b2df-132df6e29949"). InnerVolumeSpecName "kube-api-access-c646z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:17:47.492697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:47.492599 6389 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-webhook-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:17:47.492697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:47.492652 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c646z\" (UniqueName: \"kubernetes.io/projected/1859e393-7fa6-4046-b2df-132df6e29949-kube-api-access-c646z\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:17:47.492697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:47.492665 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1859e393-7fa6-4046-b2df-132df6e29949-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:17:48.017247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:48.017214 6389 generic.go:356] "Generic (PLEG): container finished" podID="1859e393-7fa6-4046-b2df-132df6e29949" containerID="1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9" exitCode=0 Nov 03 14:17:48.017699 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:48.017259 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64678f5d5d-96mcs" event={"ID":"1859e393-7fa6-4046-b2df-132df6e29949","Type":"ContainerDied","Data":"1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9"} Nov 03 14:17:48.017699 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:48.017269 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64678f5d5d-96mcs" Nov 03 14:17:48.017699 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:48.017301 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64678f5d5d-96mcs" event={"ID":"1859e393-7fa6-4046-b2df-132df6e29949","Type":"ContainerDied","Data":"b2aa6bd3e4f171bfe190627b414c7311028b3425ac25716e9e81fa9abc73b6e4"} Nov 03 14:17:48.017699 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:48.017321 6389 scope.go:122] "RemoveContainer" containerID="1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9" Nov 03 14:17:48.025785 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:48.025767 6389 scope.go:122] "RemoveContainer" containerID="1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9" Nov 03 14:17:48.026064 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:17:48.026047 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9\": container with ID starting with 1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9 not found: ID does not exist" containerID="1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9" Nov 03 14:17:48.026114 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:48.026073 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9"} err="failed to get container status \"1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9\": rpc error: code = NotFound desc = could not find container \"1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9\": container with ID starting with 1fcdd8503a232861f6953de645bbf0ba2f2af8f4cb45a4f18f06b81bc2fb15f9 not found: ID does not exist" Nov 03 14:17:48.067329 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:48.067288 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64678f5d5d-96mcs"] Nov 03 14:17:48.078558 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:48.078524 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64678f5d5d-96mcs"] Nov 03 14:17:48.616260 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:17:48.616227 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1859e393-7fa6-4046-b2df-132df6e29949" path="/var/lib/kubelet/pods/1859e393-7fa6-4046-b2df-132df6e29949/volumes" Nov 03 14:17:58.372629 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:17:58.372576 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:18:10.517051 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.517011 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9"] Nov 03 14:18:10.517570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.517549 6389 cpu_manager.go:414] "containerMap: removing container" podUID="1859e393-7fa6-4046-b2df-132df6e29949" containerName="manager" Nov 03 14:18:10.517685 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.517573 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1859e393-7fa6-4046-b2df-132df6e29949" containerName="manager" Nov 03 14:18:10.517685 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.517598 6389 cpu_manager.go:414] "containerMap: removing container" podUID="90f82a42-a4ff-43dc-96c5-56135f30576a" containerName="discovery" Nov 03 14:18:10.517685 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.517623 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f82a42-a4ff-43dc-96c5-56135f30576a" containerName="discovery" Nov 03 14:18:10.517847 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.517718 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f82a42-a4ff-43dc-96c5-56135f30576a" containerName="discovery" Nov 03 14:18:10.517847 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.517738 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="1859e393-7fa6-4046-b2df-132df6e29949" containerName="manager" Nov 03 14:18:10.535261 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.535228 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.540828 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.540802 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9"] Nov 03 14:18:10.618857 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.618831 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 14:18:10.618857 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.618846 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cpjn\"" Nov 03 14:18:10.619059 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.618903 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 14:18:10.696126 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.696078 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngbpr\" (UniqueName: \"kubernetes.io/projected/dfd6c738-f2a8-4755-8266-e9b7d966c915-kube-api-access-ngbpr\") pod \"maas-api-749c7b6d78-42bx9\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.696126 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.696124 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-metrics-certs\") pod \"maas-api-749c7b6d78-42bx9\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.696342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.696172 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-maas-api-tls\") pod \"maas-api-749c7b6d78-42bx9\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.696342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.696327 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd6c738-f2a8-4755-8266-e9b7d966c915-openshift-service-ca\") pod \"maas-api-749c7b6d78-42bx9\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.797843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.797758 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngbpr\" (UniqueName: \"kubernetes.io/projected/dfd6c738-f2a8-4755-8266-e9b7d966c915-kube-api-access-ngbpr\") pod \"maas-api-749c7b6d78-42bx9\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.797843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.797810 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-metrics-certs\") pod \"maas-api-749c7b6d78-42bx9\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.797843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.797844 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-maas-api-tls\") pod \"maas-api-749c7b6d78-42bx9\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.798141 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.797947 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd6c738-f2a8-4755-8266-e9b7d966c915-openshift-service-ca\") pod \"maas-api-749c7b6d78-42bx9\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.798683 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.798624 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd6c738-f2a8-4755-8266-e9b7d966c915-openshift-service-ca\") pod \"maas-api-749c7b6d78-42bx9\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.800412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.800391 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-maas-api-tls\") pod \"maas-api-749c7b6d78-42bx9\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.800476 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.800408 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-metrics-certs\") pod \"maas-api-749c7b6d78-42bx9\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.817984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.817960 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngbpr\" (UniqueName: \"kubernetes.io/projected/dfd6c738-f2a8-4755-8266-e9b7d966c915-kube-api-access-ngbpr\") pod \"maas-api-749c7b6d78-42bx9\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:10.850554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:10.850524 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:11.150508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:11.150426 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9"] Nov 03 14:18:11.153227 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:18:11.153200 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd6c738_f2a8_4755_8266_e9b7d966c915.slice/crio-b2d37c1c17bb5a5cb4db927594c7fb0e9e877008aef941f65eba213d49879c45 WatchSource:0}: Error finding container b2d37c1c17bb5a5cb4db927594c7fb0e9e877008aef941f65eba213d49879c45: Status 404 returned error can't find the container with id b2d37c1c17bb5a5cb4db927594c7fb0e9e877008aef941f65eba213d49879c45 Nov 03 14:18:12.118786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:12.118745 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" event={"ID":"dfd6c738-f2a8-4755-8266-e9b7d966c915","Type":"ContainerStarted","Data":"b2d37c1c17bb5a5cb4db927594c7fb0e9e877008aef941f65eba213d49879c45"} Nov 03 14:18:13.124210 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:13.124173 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" event={"ID":"dfd6c738-f2a8-4755-8266-e9b7d966c915","Type":"ContainerStarted","Data":"cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9"} Nov 03 14:18:13.124789 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:13.124324 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:13.164793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:13.164737 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" podStartSLOduration=1.326906243 podStartE2EDuration="3.164722985s" totalImagesPullingTime="1.837816742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:18:13.164407436 +0000 UTC m=+368.723043923" watchObservedRunningTime="2026-11-03 14:18:13.164722985 +0000 UTC m=+368.723359468" Nov 03 14:18:19.133832 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:19.133804 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:20.079120 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.079082 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5757765976-lklsf"] Nov 03 14:18:20.082704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.082685 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.113521 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.113495 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5757765976-lklsf"] Nov 03 14:18:20.170974 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.170925 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-maas-api-tls\") pod \"maas-api-5757765976-lklsf\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.171391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.171044 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-metrics-certs\") pod \"maas-api-5757765976-lklsf\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.171391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.171105 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbvt7\" (UniqueName: \"kubernetes.io/projected/466189f1-5bf9-4bb4-8eee-874a7407bc8a-kube-api-access-vbvt7\") pod \"maas-api-5757765976-lklsf\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.171391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.171136 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466189f1-5bf9-4bb4-8eee-874a7407bc8a-openshift-service-ca\") pod \"maas-api-5757765976-lklsf\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.272535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.272492 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-metrics-certs\") pod \"maas-api-5757765976-lklsf\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.272535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.272527 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbvt7\" (UniqueName: \"kubernetes.io/projected/466189f1-5bf9-4bb4-8eee-874a7407bc8a-kube-api-access-vbvt7\") pod \"maas-api-5757765976-lklsf\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.273398 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.272552 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466189f1-5bf9-4bb4-8eee-874a7407bc8a-openshift-service-ca\") pod \"maas-api-5757765976-lklsf\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.273398 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.272627 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-maas-api-tls\") pod \"maas-api-5757765976-lklsf\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.273398 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.273335 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466189f1-5bf9-4bb4-8eee-874a7407bc8a-openshift-service-ca\") pod \"maas-api-5757765976-lklsf\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.275216 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.275182 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-maas-api-tls\") pod \"maas-api-5757765976-lklsf\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.275338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.275224 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-metrics-certs\") pod \"maas-api-5757765976-lklsf\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.292432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.292404 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbvt7\" (UniqueName: \"kubernetes.io/projected/466189f1-5bf9-4bb4-8eee-874a7407bc8a-kube-api-access-vbvt7\") pod \"maas-api-5757765976-lklsf\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.395239 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.395154 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:20.696222 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:20.696196 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5757765976-lklsf"] Nov 03 14:18:20.699268 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:18:20.699226 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466189f1_5bf9_4bb4_8eee_874a7407bc8a.slice/crio-8df03a24051560694724064ebb10da10cd50a748b36aba83c60361b8eafcdb9d WatchSource:0}: Error finding container 8df03a24051560694724064ebb10da10cd50a748b36aba83c60361b8eafcdb9d: Status 404 returned error can't find the container with id 8df03a24051560694724064ebb10da10cd50a748b36aba83c60361b8eafcdb9d Nov 03 14:18:21.157567 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:21.157533 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" event={"ID":"466189f1-5bf9-4bb4-8eee-874a7407bc8a","Type":"ContainerStarted","Data":"91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2"} Nov 03 14:18:21.157797 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:21.157583 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" event={"ID":"466189f1-5bf9-4bb4-8eee-874a7407bc8a","Type":"ContainerStarted","Data":"8df03a24051560694724064ebb10da10cd50a748b36aba83c60361b8eafcdb9d"} Nov 03 14:18:21.157797 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:21.157673 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:21.198244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:21.198196 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" podStartSLOduration=0.853445611 podStartE2EDuration="1.198183051s" totalImagesPullingTime="344.73744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:18:21.198077764 +0000 UTC m=+376.756714250" watchObservedRunningTime="2026-11-03 14:18:21.198183051 +0000 UTC m=+376.756819535" Nov 03 14:18:27.166947 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.166913 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 14:18:27.246945 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.246917 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9"] Nov 03 14:18:27.247185 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.247138 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" podUID="dfd6c738-f2a8-4755-8266-e9b7d966c915" containerName="maas-api" containerID="cri-o://cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9" gracePeriod=30 Nov 03 14:18:27.486469 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.486445 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:27.536909 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.536882 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd6c738-f2a8-4755-8266-e9b7d966c915-kube-api-access-ngbpr\" (UniqueName: \"kubernetes.io/projected/dfd6c738-f2a8-4755-8266-e9b7d966c915-kube-api-access-ngbpr\") pod \"dfd6c738-f2a8-4755-8266-e9b7d966c915\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " Nov 03 14:18:27.537108 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.536920 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-metrics-certs\") pod \"dfd6c738-f2a8-4755-8266-e9b7d966c915\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " Nov 03 14:18:27.537108 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.537024 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd6c738-f2a8-4755-8266-e9b7d966c915-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd6c738-f2a8-4755-8266-e9b7d966c915-openshift-service-ca\") pod \"dfd6c738-f2a8-4755-8266-e9b7d966c915\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " Nov 03 14:18:27.537108 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.537100 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-maas-api-tls\") pod \"dfd6c738-f2a8-4755-8266-e9b7d966c915\" (UID: \"dfd6c738-f2a8-4755-8266-e9b7d966c915\") " Nov 03 14:18:27.537401 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.537371 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd6c738-f2a8-4755-8266-e9b7d966c915-openshift-service-ca" pod "dfd6c738-f2a8-4755-8266-e9b7d966c915" (UID: "dfd6c738-f2a8-4755-8266-e9b7d966c915"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:18:27.539386 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.539354 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-metrics-certs" pod "dfd6c738-f2a8-4755-8266-e9b7d966c915" (UID: "dfd6c738-f2a8-4755-8266-e9b7d966c915"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:18:27.539505 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.539401 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-maas-api-tls" pod "dfd6c738-f2a8-4755-8266-e9b7d966c915" (UID: "dfd6c738-f2a8-4755-8266-e9b7d966c915"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:18:27.539505 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.539493 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd6c738-f2a8-4755-8266-e9b7d966c915-kube-api-access-ngbpr" pod "dfd6c738-f2a8-4755-8266-e9b7d966c915" (UID: "dfd6c738-f2a8-4755-8266-e9b7d966c915"). InnerVolumeSpecName "kube-api-access-ngbpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:18:27.638375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.638340 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:18:27.638375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.638372 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngbpr\" (UniqueName: \"kubernetes.io/projected/dfd6c738-f2a8-4755-8266-e9b7d966c915-kube-api-access-ngbpr\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:18:27.638375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.638382 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd6c738-f2a8-4755-8266-e9b7d966c915-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:18:27.638636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:27.638397 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd6c738-f2a8-4755-8266-e9b7d966c915-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:18:28.186587 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:28.186557 6389 generic.go:356] "Generic (PLEG): container finished" podID="dfd6c738-f2a8-4755-8266-e9b7d966c915" containerID="cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9" exitCode=0 Nov 03 14:18:28.187003 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:28.186625 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" Nov 03 14:18:28.187003 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:28.186649 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" event={"ID":"dfd6c738-f2a8-4755-8266-e9b7d966c915","Type":"ContainerDied","Data":"cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9"} Nov 03 14:18:28.187003 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:28.186709 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9" event={"ID":"dfd6c738-f2a8-4755-8266-e9b7d966c915","Type":"ContainerDied","Data":"b2d37c1c17bb5a5cb4db927594c7fb0e9e877008aef941f65eba213d49879c45"} Nov 03 14:18:28.187003 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:28.186738 6389 scope.go:122] "RemoveContainer" containerID="cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9" Nov 03 14:18:28.196005 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:28.195988 6389 scope.go:122] "RemoveContainer" containerID="cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9" Nov 03 14:18:28.196273 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:18:28.196257 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9\": container with ID starting with cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9 not found: ID does not exist" containerID="cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9" Nov 03 14:18:28.196310 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:28.196282 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9"} err="failed to get container status \"cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9\": rpc error: code = NotFound desc = could not find container \"cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9\": container with ID starting with cad066d4f111945b2e231b218b07a685fa89ff5a9e2d58a8ae5ebc49ed6937d9 not found: ID does not exist" Nov 03 14:18:28.236860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:28.236832 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9"] Nov 03 14:18:28.248089 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:28.248070 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-749c7b6d78-42bx9"] Nov 03 14:18:28.372692 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:18:28.372659 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:18:28.615572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:18:28.615501 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd6c738-f2a8-4755-8266-e9b7d966c915" path="/var/lib/kubelet/pods/dfd6c738-f2a8-4755-8266-e9b7d966c915/volumes" Nov 03 14:18:58.373379 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:18:58.373339 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:19:17.878181 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:17.878140 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-wzdx8"] Nov 03 14:19:17.878635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:17.878471 6389 cpu_manager.go:414] "containerMap: removing container" podUID="dfd6c738-f2a8-4755-8266-e9b7d966c915" containerName="maas-api" Nov 03 14:19:17.878635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:17.878482 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd6c738-f2a8-4755-8266-e9b7d966c915" containerName="maas-api" Nov 03 14:19:17.878635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:17.878557 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd6c738-f2a8-4755-8266-e9b7d966c915" containerName="maas-api" Nov 03 14:19:17.881575 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:17.881559 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" Nov 03 14:19:17.900875 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:17.900843 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 14:19:17.902116 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:17.902078 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-wzdx8"] Nov 03 14:19:17.912291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:17.912266 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-m8jt5\"" Nov 03 14:19:17.929814 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:17.929790 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 14:19:17.952417 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:17.952385 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621821ff-8102-47a3-8258-07021d449e13-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-wzdx8\" (UID: \"621821ff-8102-47a3-8258-07021d449e13\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" Nov 03 14:19:17.952566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:17.952545 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4scps\" (UniqueName: \"kubernetes.io/projected/621821ff-8102-47a3-8258-07021d449e13-kube-api-access-4scps\") pod \"ai-gateway-controller-5d8dddb676-wzdx8\" (UID: \"621821ff-8102-47a3-8258-07021d449e13\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" Nov 03 14:19:18.053516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:18.053484 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4scps\" (UniqueName: \"kubernetes.io/projected/621821ff-8102-47a3-8258-07021d449e13-kube-api-access-4scps\") pod \"ai-gateway-controller-5d8dddb676-wzdx8\" (UID: \"621821ff-8102-47a3-8258-07021d449e13\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" Nov 03 14:19:18.053723 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:18.053532 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621821ff-8102-47a3-8258-07021d449e13-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-wzdx8\" (UID: \"621821ff-8102-47a3-8258-07021d449e13\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" Nov 03 14:19:18.056025 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:18.056003 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621821ff-8102-47a3-8258-07021d449e13-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-wzdx8\" (UID: \"621821ff-8102-47a3-8258-07021d449e13\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" Nov 03 14:19:18.073794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:18.073766 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4scps\" (UniqueName: \"kubernetes.io/projected/621821ff-8102-47a3-8258-07021d449e13-kube-api-access-4scps\") pod \"ai-gateway-controller-5d8dddb676-wzdx8\" (UID: \"621821ff-8102-47a3-8258-07021d449e13\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" Nov 03 14:19:18.193929 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:18.193898 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" Nov 03 14:19:18.490155 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:18.490132 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-wzdx8"] Nov 03 14:19:18.492645 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:19:18.492620 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621821ff_8102_47a3_8258_07021d449e13.slice/crio-91c1f79ba78c37bc4a82be93e61d067940b284a641dbb18502b0e5606b0c72f5 WatchSource:0}: Error finding container 91c1f79ba78c37bc4a82be93e61d067940b284a641dbb18502b0e5606b0c72f5: Status 404 returned error can't find the container with id 91c1f79ba78c37bc4a82be93e61d067940b284a641dbb18502b0e5606b0c72f5 Nov 03 14:19:19.396492 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:19.396455 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" event={"ID":"621821ff-8102-47a3-8258-07021d449e13","Type":"ContainerStarted","Data":"91c1f79ba78c37bc4a82be93e61d067940b284a641dbb18502b0e5606b0c72f5"} Nov 03 14:19:21.406201 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:21.406161 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" event={"ID":"621821ff-8102-47a3-8258-07021d449e13","Type":"ContainerStarted","Data":"3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062"} Nov 03 14:19:21.406599 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:21.406282 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" Nov 03 14:19:21.447535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:21.447470 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" podStartSLOduration=2.463389605 podStartE2EDuration="4.447453777s" totalImagesPullingTime="1.984064172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:19:21.446296407 +0000 UTC m=+437.004932890" watchObservedRunningTime="2026-11-03 14:19:21.447453777 +0000 UTC m=+437.006090260" Nov 03 14:19:23.093979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.093951 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn"] Nov 03 14:19:23.097689 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.097667 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" Nov 03 14:19:23.113465 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.113443 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 14:19:23.123350 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.123326 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-wrnr9\"" Nov 03 14:19:23.124281 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.124259 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn"] Nov 03 14:19:23.179758 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.179720 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r9bfq"] Nov 03 14:19:23.183110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.183085 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" Nov 03 14:19:23.202867 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.202836 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfzm\" (UniqueName: \"kubernetes.io/projected/a41e843c-6800-4404-a7cb-adb5af6493f9-kube-api-access-6sfzm\") pod \"payload-processing-55fbbbf9dc-r9bfq\" (UID: \"a41e843c-6800-4404-a7cb-adb5af6493f9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" Nov 03 14:19:23.203086 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.203065 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt88n\" (UniqueName: \"kubernetes.io/projected/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-kube-api-access-jt88n\") pod \"payload-pre-processing-7b98bb678c-b9jkn\" (UID: \"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" Nov 03 14:19:23.203185 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.203101 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41e843c-6800-4404-a7cb-adb5af6493f9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-r9bfq\" (UID: \"a41e843c-6800-4404-a7cb-adb5af6493f9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" Nov 03 14:19:23.203185 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.203127 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-b9jkn\" (UID: \"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" Nov 03 14:19:23.204186 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.204162 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r9bfq"] Nov 03 14:19:23.304300 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.304272 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfzm\" (UniqueName: \"kubernetes.io/projected/a41e843c-6800-4404-a7cb-adb5af6493f9-kube-api-access-6sfzm\") pod \"payload-processing-55fbbbf9dc-r9bfq\" (UID: \"a41e843c-6800-4404-a7cb-adb5af6493f9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" Nov 03 14:19:23.304488 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.304320 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt88n\" (UniqueName: \"kubernetes.io/projected/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-kube-api-access-jt88n\") pod \"payload-pre-processing-7b98bb678c-b9jkn\" (UID: \"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" Nov 03 14:19:23.304488 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.304351 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41e843c-6800-4404-a7cb-adb5af6493f9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-r9bfq\" (UID: \"a41e843c-6800-4404-a7cb-adb5af6493f9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" Nov 03 14:19:23.304488 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.304384 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-b9jkn\" (UID: \"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" Nov 03 14:19:23.305120 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.305102 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41e843c-6800-4404-a7cb-adb5af6493f9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-r9bfq\" (UID: \"a41e843c-6800-4404-a7cb-adb5af6493f9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" Nov 03 14:19:23.305160 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.305120 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-b9jkn\" (UID: \"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" Nov 03 14:19:23.326867 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.326833 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt88n\" (UniqueName: \"kubernetes.io/projected/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-kube-api-access-jt88n\") pod \"payload-pre-processing-7b98bb678c-b9jkn\" (UID: \"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" Nov 03 14:19:23.329750 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.329719 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfzm\" (UniqueName: \"kubernetes.io/projected/a41e843c-6800-4404-a7cb-adb5af6493f9-kube-api-access-6sfzm\") pod \"payload-processing-55fbbbf9dc-r9bfq\" (UID: \"a41e843c-6800-4404-a7cb-adb5af6493f9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" Nov 03 14:19:23.410469 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.410382 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" Nov 03 14:19:23.496103 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.496072 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" Nov 03 14:19:23.707395 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.707354 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn"] Nov 03 14:19:23.710219 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:19:23.710194 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f57c9a_972c_49a8_a1d8_b703dd4d02a4.slice/crio-5bfcf7dd370b0436639600709f6c149e364f25215be0f826ddccead143c7d521 WatchSource:0}: Error finding container 5bfcf7dd370b0436639600709f6c149e364f25215be0f826ddccead143c7d521: Status 404 returned error can't find the container with id 5bfcf7dd370b0436639600709f6c149e364f25215be0f826ddccead143c7d521 Nov 03 14:19:23.791539 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:23.791513 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r9bfq"] Nov 03 14:19:23.793479 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:19:23.793456 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41e843c_6800_4404_a7cb_adb5af6493f9.slice/crio-217e19de94d34f70f58872827f7f70f2271b19e64ed1b5ece59c56ecbda0adbf WatchSource:0}: Error finding container 217e19de94d34f70f58872827f7f70f2271b19e64ed1b5ece59c56ecbda0adbf: Status 404 returned error can't find the container with id 217e19de94d34f70f58872827f7f70f2271b19e64ed1b5ece59c56ecbda0adbf Nov 03 14:19:24.422886 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:24.422777 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" event={"ID":"a41e843c-6800-4404-a7cb-adb5af6493f9","Type":"ContainerStarted","Data":"217e19de94d34f70f58872827f7f70f2271b19e64ed1b5ece59c56ecbda0adbf"} Nov 03 14:19:24.424732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:24.424630 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" event={"ID":"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4","Type":"ContainerStarted","Data":"5bfcf7dd370b0436639600709f6c149e364f25215be0f826ddccead143c7d521"} Nov 03 14:19:26.434891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:26.434854 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" event={"ID":"a41e843c-6800-4404-a7cb-adb5af6493f9","Type":"ContainerStarted","Data":"4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c"} Nov 03 14:19:26.435348 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:26.434928 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" Nov 03 14:19:26.436410 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:26.436386 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" event={"ID":"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4","Type":"ContainerStarted","Data":"ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36"} Nov 03 14:19:26.436553 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:26.436540 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" Nov 03 14:19:26.478247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:26.478181 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" podStartSLOduration=1.238613348 podStartE2EDuration="3.478163558s" totalImagesPullingTime="2.23955021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:19:26.476289703 +0000 UTC m=+442.034926211" watchObservedRunningTime="2026-11-03 14:19:26.478163558 +0000 UTC m=+442.036800071" Nov 03 14:19:26.520923 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:26.520850 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" podStartSLOduration=1.154369565 podStartE2EDuration="3.520834362s" totalImagesPullingTime="2.366464797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:19:26.519932697 +0000 UTC m=+442.078569180" watchObservedRunningTime="2026-11-03 14:19:26.520834362 +0000 UTC m=+442.079470879" Nov 03 14:19:28.373671 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:19:28.373630 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:19:32.413308 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:32.413278 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" Nov 03 14:19:32.724665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:32.724600 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f464594c6-562pp"] Nov 03 14:19:32.728178 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:32.728161 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" Nov 03 14:19:32.747633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:32.747589 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f464594c6-562pp"] Nov 03 14:19:32.786691 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:32.786660 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkchd\" (UniqueName: \"kubernetes.io/projected/428b2a8c-5dc4-4b29-a535-35b8b20cf721-kube-api-access-jkchd\") pod \"ai-gateway-controller-5f464594c6-562pp\" (UID: \"428b2a8c-5dc4-4b29-a535-35b8b20cf721\") " pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" Nov 03 14:19:32.786854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:32.786724 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428b2a8c-5dc4-4b29-a535-35b8b20cf721-metrics-certs\") pod \"ai-gateway-controller-5f464594c6-562pp\" (UID: \"428b2a8c-5dc4-4b29-a535-35b8b20cf721\") " pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" Nov 03 14:19:32.887531 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:32.887497 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkchd\" (UniqueName: \"kubernetes.io/projected/428b2a8c-5dc4-4b29-a535-35b8b20cf721-kube-api-access-jkchd\") pod \"ai-gateway-controller-5f464594c6-562pp\" (UID: \"428b2a8c-5dc4-4b29-a535-35b8b20cf721\") " pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" Nov 03 14:19:32.887745 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:32.887559 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428b2a8c-5dc4-4b29-a535-35b8b20cf721-metrics-certs\") pod \"ai-gateway-controller-5f464594c6-562pp\" (UID: \"428b2a8c-5dc4-4b29-a535-35b8b20cf721\") " pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" Nov 03 14:19:32.889969 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:32.889943 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428b2a8c-5dc4-4b29-a535-35b8b20cf721-metrics-certs\") pod \"ai-gateway-controller-5f464594c6-562pp\" (UID: \"428b2a8c-5dc4-4b29-a535-35b8b20cf721\") " pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" Nov 03 14:19:32.907914 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:32.907883 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkchd\" (UniqueName: \"kubernetes.io/projected/428b2a8c-5dc4-4b29-a535-35b8b20cf721-kube-api-access-jkchd\") pod \"ai-gateway-controller-5f464594c6-562pp\" (UID: \"428b2a8c-5dc4-4b29-a535-35b8b20cf721\") " pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" Nov 03 14:19:33.040861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:33.040784 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" Nov 03 14:19:33.337856 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:33.337803 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f464594c6-562pp"] Nov 03 14:19:33.340011 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:19:33.339985 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428b2a8c_5dc4_4b29_a535_35b8b20cf721.slice/crio-2ad24c8cad6f92ba777abff42f860bcf2b47ef1c823682ecd86dfee44b4e375c WatchSource:0}: Error finding container 2ad24c8cad6f92ba777abff42f860bcf2b47ef1c823682ecd86dfee44b4e375c: Status 404 returned error can't find the container with id 2ad24c8cad6f92ba777abff42f860bcf2b47ef1c823682ecd86dfee44b4e375c Nov 03 14:19:33.467048 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:33.467016 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" event={"ID":"428b2a8c-5dc4-4b29-a535-35b8b20cf721","Type":"ContainerStarted","Data":"2ad24c8cad6f92ba777abff42f860bcf2b47ef1c823682ecd86dfee44b4e375c"} Nov 03 14:19:34.472906 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:34.472862 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" event={"ID":"428b2a8c-5dc4-4b29-a535-35b8b20cf721","Type":"ContainerStarted","Data":"7963271c4ad7b06b40bb214aac05c85dc8213e74003422dc79602897f9291597"} Nov 03 14:19:34.473347 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:34.472939 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" Nov 03 14:19:34.513511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:34.513457 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" podStartSLOduration=2.175599035 podStartE2EDuration="2.513441911s" totalImagesPullingTime="337.842876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:19:34.512803647 +0000 UTC m=+450.071440132" watchObservedRunningTime="2026-11-03 14:19:34.513441911 +0000 UTC m=+450.072078373" Nov 03 14:19:37.444712 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:37.444675 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" Nov 03 14:19:37.445166 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:37.444857 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" Nov 03 14:19:45.478737 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:45.478662 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f464594c6-562pp" Nov 03 14:19:45.547999 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:45.547965 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-wzdx8"] Nov 03 14:19:45.548250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:45.548200 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" podUID="621821ff-8102-47a3-8258-07021d449e13" containerName="manager" containerID="cri-o://3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062" gracePeriod=30 Nov 03 14:19:45.787470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:45.787446 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" Nov 03 14:19:45.901820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:45.901790 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621821ff-8102-47a3-8258-07021d449e13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/621821ff-8102-47a3-8258-07021d449e13-metrics-certs\") pod \"621821ff-8102-47a3-8258-07021d449e13\" (UID: \"621821ff-8102-47a3-8258-07021d449e13\") " Nov 03 14:19:45.902001 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:45.901882 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621821ff-8102-47a3-8258-07021d449e13-kube-api-access-4scps\" (UniqueName: \"kubernetes.io/projected/621821ff-8102-47a3-8258-07021d449e13-kube-api-access-4scps\") pod \"621821ff-8102-47a3-8258-07021d449e13\" (UID: \"621821ff-8102-47a3-8258-07021d449e13\") " Nov 03 14:19:45.903972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:45.903944 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621821ff-8102-47a3-8258-07021d449e13-metrics-certs" pod "621821ff-8102-47a3-8258-07021d449e13" (UID: "621821ff-8102-47a3-8258-07021d449e13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:19:45.903972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:45.903962 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621821ff-8102-47a3-8258-07021d449e13-kube-api-access-4scps" pod "621821ff-8102-47a3-8258-07021d449e13" (UID: "621821ff-8102-47a3-8258-07021d449e13"). InnerVolumeSpecName "kube-api-access-4scps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:46.002466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:46.002432 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4scps\" (UniqueName: \"kubernetes.io/projected/621821ff-8102-47a3-8258-07021d449e13-kube-api-access-4scps\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:19:46.002466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:46.002462 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621821ff-8102-47a3-8258-07021d449e13-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:19:46.521018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:46.520976 6389 generic.go:356] "Generic (PLEG): container finished" podID="621821ff-8102-47a3-8258-07021d449e13" containerID="3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062" exitCode=0 Nov 03 14:19:46.521515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:46.521036 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" Nov 03 14:19:46.521515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:46.521064 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" event={"ID":"621821ff-8102-47a3-8258-07021d449e13","Type":"ContainerDied","Data":"3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062"} Nov 03 14:19:46.521515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:46.521140 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-wzdx8" event={"ID":"621821ff-8102-47a3-8258-07021d449e13","Type":"ContainerDied","Data":"91c1f79ba78c37bc4a82be93e61d067940b284a641dbb18502b0e5606b0c72f5"} Nov 03 14:19:46.521515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:46.521163 6389 scope.go:122] "RemoveContainer" containerID="3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062" Nov 03 14:19:46.530325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:46.530306 6389 scope.go:122] "RemoveContainer" containerID="3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062" Nov 03 14:19:46.530630 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:19:46.530588 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062\": container with ID starting with 3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062 not found: ID does not exist" containerID="3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062" Nov 03 14:19:46.530713 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:46.530633 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062"} err="failed to get container status \"3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062\": rpc error: code = NotFound desc = could not find container \"3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062\": container with ID starting with 3b1b0506dbabd8a07366c8c6866f49416fbda092d2f1651bf8ebb9a797c96062 not found: ID does not exist" Nov 03 14:19:46.572003 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:46.571975 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-wzdx8"] Nov 03 14:19:46.583185 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:46.583154 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-wzdx8"] Nov 03 14:19:46.617786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:46.617761 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621821ff-8102-47a3-8258-07021d449e13" path="/var/lib/kubelet/pods/621821ff-8102-47a3-8258-07021d449e13/volumes" Nov 03 14:19:52.668838 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.668807 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4"] Nov 03 14:19:52.669331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.669312 6389 cpu_manager.go:414] "containerMap: removing container" podUID="621821ff-8102-47a3-8258-07021d449e13" containerName="manager" Nov 03 14:19:52.669412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.669334 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621821ff-8102-47a3-8258-07021d449e13" containerName="manager" Nov 03 14:19:52.669470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.669447 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="621821ff-8102-47a3-8258-07021d449e13" containerName="manager" Nov 03 14:19:52.673955 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.673932 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.693176 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.693149 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4"] Nov 03 14:19:52.702893 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.702871 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 14:19:52.703065 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.703049 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 14:19:52.703124 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.703062 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 14:19:52.762546 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.762514 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42f57c84-c009-4eab-8029-633dc6e21aa2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.762731 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.762560 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f57c84-c009-4eab-8029-633dc6e21aa2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.762731 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.762592 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42f57c84-c009-4eab-8029-633dc6e21aa2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.762731 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.762694 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42f57c84-c009-4eab-8029-633dc6e21aa2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.762731 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.762711 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42f57c84-c009-4eab-8029-633dc6e21aa2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.762902 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.762737 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vnjn\" (UniqueName: \"kubernetes.io/projected/42f57c84-c009-4eab-8029-633dc6e21aa2-kube-api-access-5vnjn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.792558 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.792531 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dhg2d\"" Nov 03 14:19:52.863181 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.863140 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f57c84-c009-4eab-8029-633dc6e21aa2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.863363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.863205 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42f57c84-c009-4eab-8029-633dc6e21aa2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.863363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.863248 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42f57c84-c009-4eab-8029-633dc6e21aa2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.863363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.863274 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42f57c84-c009-4eab-8029-633dc6e21aa2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.863363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.863312 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vnjn\" (UniqueName: \"kubernetes.io/projected/42f57c84-c009-4eab-8029-633dc6e21aa2-kube-api-access-5vnjn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.863363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.863354 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42f57c84-c009-4eab-8029-633dc6e21aa2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.863779 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.863751 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42f57c84-c009-4eab-8029-633dc6e21aa2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.863779 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.863765 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42f57c84-c009-4eab-8029-633dc6e21aa2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.863920 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.863797 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42f57c84-c009-4eab-8029-633dc6e21aa2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.865718 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.865698 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42f57c84-c009-4eab-8029-633dc6e21aa2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.865897 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.865881 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f57c84-c009-4eab-8029-633dc6e21aa2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.885726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.885702 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vnjn\" (UniqueName: \"kubernetes.io/projected/42f57c84-c009-4eab-8029-633dc6e21aa2-kube-api-access-5vnjn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4\" (UID: \"42f57c84-c009-4eab-8029-633dc6e21aa2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:52.986761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:52.986674 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:53.292094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:53.292067 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4"] Nov 03 14:19:53.294274 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:19:53.294250 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f57c84_c009_4eab_8029_633dc6e21aa2.slice/crio-73f911c393d78d8d7b5fe5e3f259d24866a37cac51feb5104f35d796af6de8c3 WatchSource:0}: Error finding container 73f911c393d78d8d7b5fe5e3f259d24866a37cac51feb5104f35d796af6de8c3: Status 404 returned error can't find the container with id 73f911c393d78d8d7b5fe5e3f259d24866a37cac51feb5104f35d796af6de8c3 Nov 03 14:19:53.554317 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:53.554230 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" event={"ID":"42f57c84-c009-4eab-8029-633dc6e21aa2","Type":"ContainerStarted","Data":"73f911c393d78d8d7b5fe5e3f259d24866a37cac51feb5104f35d796af6de8c3"} Nov 03 14:19:55.564553 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.564512 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" event={"ID":"42f57c84-c009-4eab-8029-633dc6e21aa2","Type":"ContainerStarted","Data":"e752be1d48843b3597be2e94faa067ba88bc8130ba6187a2885f3916ce6837a7"} Nov 03 14:19:55.607802 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.607749 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" podStartSLOduration=2.28360387 podStartE2EDuration="3.607730535s" totalImagesPullingTime="1.324126665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:19:55.606798656 +0000 UTC m=+471.165435141" watchObservedRunningTime="2026-11-03 14:19:55.607730535 +0000 UTC m=+471.166367050" Nov 03 14:19:55.778126 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.778093 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr"] Nov 03 14:19:55.781997 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.781969 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.797348 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.797326 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 14:19:55.801389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.801366 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr"] Nov 03 14:19:55.893812 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.893731 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87e0fff-43c1-4030-b7bc-9c16a38e6872-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.893812 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.893795 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64jt4\" (UniqueName: \"kubernetes.io/projected/d87e0fff-43c1-4030-b7bc-9c16a38e6872-kube-api-access-64jt4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.893997 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.893872 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87e0fff-43c1-4030-b7bc-9c16a38e6872-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.893997 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.893899 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d87e0fff-43c1-4030-b7bc-9c16a38e6872-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.893997 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.893929 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d87e0fff-43c1-4030-b7bc-9c16a38e6872-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.894109 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.894002 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87e0fff-43c1-4030-b7bc-9c16a38e6872-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.987525 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.987496 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:19:55.994389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.994362 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87e0fff-43c1-4030-b7bc-9c16a38e6872-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.994554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.994407 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87e0fff-43c1-4030-b7bc-9c16a38e6872-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.994554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.994439 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64jt4\" (UniqueName: \"kubernetes.io/projected/d87e0fff-43c1-4030-b7bc-9c16a38e6872-kube-api-access-64jt4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.994554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.994465 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87e0fff-43c1-4030-b7bc-9c16a38e6872-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.994554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.994483 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d87e0fff-43c1-4030-b7bc-9c16a38e6872-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.994794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.994601 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d87e0fff-43c1-4030-b7bc-9c16a38e6872-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.994996 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.994973 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d87e0fff-43c1-4030-b7bc-9c16a38e6872-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.995068 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.994974 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87e0fff-43c1-4030-b7bc-9c16a38e6872-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.995134 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.995103 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87e0fff-43c1-4030-b7bc-9c16a38e6872-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.996798 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.996769 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87e0fff-43c1-4030-b7bc-9c16a38e6872-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:55.997148 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:55.997129 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d87e0fff-43c1-4030-b7bc-9c16a38e6872-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:56.014688 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:56.014664 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64jt4\" (UniqueName: \"kubernetes.io/projected/d87e0fff-43c1-4030-b7bc-9c16a38e6872-kube-api-access-64jt4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d77tr\" (UID: \"d87e0fff-43c1-4030-b7bc-9c16a38e6872\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:56.095118 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:56.095076 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:56.394005 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:56.393932 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr"] Nov 03 14:19:56.396565 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:19:56.396535 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87e0fff_43c1_4030_b7bc_9c16a38e6872.slice/crio-653cf0e331cbe3c7ec69f8e55e8784c81df832d197b8f9f6a2804d6640fccc5b WatchSource:0}: Error finding container 653cf0e331cbe3c7ec69f8e55e8784c81df832d197b8f9f6a2804d6640fccc5b: Status 404 returned error can't find the container with id 653cf0e331cbe3c7ec69f8e55e8784c81df832d197b8f9f6a2804d6640fccc5b Nov 03 14:19:56.570509 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:56.570471 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" event={"ID":"d87e0fff-43c1-4030-b7bc-9c16a38e6872","Type":"ContainerStarted","Data":"653cf0e331cbe3c7ec69f8e55e8784c81df832d197b8f9f6a2804d6640fccc5b"} Nov 03 14:19:57.576430 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:57.576390 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" event={"ID":"d87e0fff-43c1-4030-b7bc-9c16a38e6872","Type":"ContainerStarted","Data":"e207559b40d00a4d8b9861c2559ac002c74932bd99679449655f3d586680eabf"} Nov 03 14:19:57.616191 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:57.616133 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" podStartSLOduration=2.393923071 podStartE2EDuration="2.616120162s" totalImagesPullingTime="222.197091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:19:57.615670664 +0000 UTC m=+473.174307147" watchObservedRunningTime="2026-11-03 14:19:57.616120162 +0000 UTC m=+473.174756645" Nov 03 14:19:58.095500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:58.095465 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:19:58.376758 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:19:58.376673 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:19:59.641527 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.641498 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n"] Nov 03 14:19:59.648156 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.648129 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.670368 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.670338 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n"] Nov 03 14:19:59.677870 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.677850 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 14:19:59.728969 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.728938 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.729133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.728983 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bng6\" (UniqueName: \"kubernetes.io/projected/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-kube-api-access-4bng6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.729205 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.729137 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.729267 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.729243 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.729322 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.729270 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.729377 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.729362 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.830370 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.830338 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.830370 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.830378 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.830636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.830398 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.830636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.830430 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.830636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.830465 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bng6\" (UniqueName: \"kubernetes.io/projected/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-kube-api-access-4bng6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.830636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.830537 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.830882 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.830852 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.831100 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.831073 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.831212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.831116 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.832869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.832847 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.833110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.833093 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.852548 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.852519 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bng6\" (UniqueName: \"kubernetes.io/projected/35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2-kube-api-access-4bng6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n\" (UID: \"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:19:59.961706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:19:59.961674 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:20:00.266108 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:00.266084 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n"] Nov 03 14:20:00.268467 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:20:00.268440 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35993ab8_aaa0_41cd_b4cf_5f0e85fb11d2.slice/crio-34875cbef806e976a2c10456fa6dfc65d355ace59ae704b92129e7d18478694d WatchSource:0}: Error finding container 34875cbef806e976a2c10456fa6dfc65d355ace59ae704b92129e7d18478694d: Status 404 returned error can't find the container with id 34875cbef806e976a2c10456fa6dfc65d355ace59ae704b92129e7d18478694d Nov 03 14:20:00.590954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:00.590926 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" event={"ID":"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2","Type":"ContainerStarted","Data":"34875cbef806e976a2c10456fa6dfc65d355ace59ae704b92129e7d18478694d"} Nov 03 14:20:01.158530 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.158500 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-c242s"] Nov 03 14:20:01.162089 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.162062 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.183727 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.183701 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-c242s"] Nov 03 14:20:01.190455 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.190432 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 14:20:01.242353 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.242317 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d2a960-6069-4a49-a738-e07f9178d7a0-home\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.242506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.242364 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d2a960-6069-4a49-a738-e07f9178d7a0-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.242506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.242491 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zrb\" (UniqueName: \"kubernetes.io/projected/d3d2a960-6069-4a49-a738-e07f9178d7a0-kube-api-access-99zrb\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.242632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.242532 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d2a960-6069-4a49-a738-e07f9178d7a0-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.242685 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.242668 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d2a960-6069-4a49-a738-e07f9178d7a0-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.242730 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.242718 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d2a960-6069-4a49-a738-e07f9178d7a0-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.343704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.343675 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d2a960-6069-4a49-a738-e07f9178d7a0-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.343864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.343715 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d2a960-6069-4a49-a738-e07f9178d7a0-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.343864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.343738 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d2a960-6069-4a49-a738-e07f9178d7a0-home\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.343864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.343782 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d2a960-6069-4a49-a738-e07f9178d7a0-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.343864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.343833 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99zrb\" (UniqueName: \"kubernetes.io/projected/d3d2a960-6069-4a49-a738-e07f9178d7a0-kube-api-access-99zrb\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.343864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.343864 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d2a960-6069-4a49-a738-e07f9178d7a0-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.344188 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.344164 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d2a960-6069-4a49-a738-e07f9178d7a0-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.344265 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.344195 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d2a960-6069-4a49-a738-e07f9178d7a0-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.344265 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.344159 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d2a960-6069-4a49-a738-e07f9178d7a0-home\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.346231 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.346211 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d2a960-6069-4a49-a738-e07f9178d7a0-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.346381 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.346364 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d2a960-6069-4a49-a738-e07f9178d7a0-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.364214 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.364188 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zrb\" (UniqueName: \"kubernetes.io/projected/d3d2a960-6069-4a49-a738-e07f9178d7a0-kube-api-access-99zrb\") pod \"echo-simulator-kserve-64b6f8f96c-c242s\" (UID: \"d3d2a960-6069-4a49-a738-e07f9178d7a0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.477465 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.477423 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:01.596945 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.596903 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" event={"ID":"35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2","Type":"ContainerStarted","Data":"c5eef441aab8965ec2cb48d9e515ee1c971bb209204bce435ac477efb3cd5ffc"} Nov 03 14:20:01.638941 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.638888 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" podStartSLOduration=2.363017579 podStartE2EDuration="2.638871768s" totalImagesPullingTime="275.854189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:01.638035286 +0000 UTC m=+477.196671771" watchObservedRunningTime="2026-11-03 14:20:01.638871768 +0000 UTC m=+477.197508251" Nov 03 14:20:01.775275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.775254 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-c242s"] Nov 03 14:20:01.777851 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:20:01.777825 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d2a960_6069_4a49_a738_e07f9178d7a0.slice/crio-b23896b49ee33bf5f197298699c1ae670a4f7c8f53324ef01e561e8f329152e6 WatchSource:0}: Error finding container b23896b49ee33bf5f197298699c1ae670a4f7c8f53324ef01e561e8f329152e6: Status 404 returned error can't find the container with id b23896b49ee33bf5f197298699c1ae670a4f7c8f53324ef01e561e8f329152e6 Nov 03 14:20:01.962033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:01.961997 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:20:02.171787 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.171759 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9"] Nov 03 14:20:02.175515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.175492 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.196515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.196490 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9"] Nov 03 14:20:02.203141 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.203116 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 14:20:02.352016 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.351987 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f9d65ca-012a-4efa-a43b-2a11844f241e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.352198 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.352039 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f9d65ca-012a-4efa-a43b-2a11844f241e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.352198 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.352117 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f9d65ca-012a-4efa-a43b-2a11844f241e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.352198 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.352162 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9d65ca-012a-4efa-a43b-2a11844f241e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.352325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.352239 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2np\" (UniqueName: \"kubernetes.io/projected/5f9d65ca-012a-4efa-a43b-2a11844f241e-kube-api-access-nv2np\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.352325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.352289 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f9d65ca-012a-4efa-a43b-2a11844f241e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.453822 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.453737 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2np\" (UniqueName: \"kubernetes.io/projected/5f9d65ca-012a-4efa-a43b-2a11844f241e-kube-api-access-nv2np\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.453822 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.453802 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f9d65ca-012a-4efa-a43b-2a11844f241e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.454042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.453849 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f9d65ca-012a-4efa-a43b-2a11844f241e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.454042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.453905 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f9d65ca-012a-4efa-a43b-2a11844f241e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.454042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.453934 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f9d65ca-012a-4efa-a43b-2a11844f241e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.454042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.453960 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9d65ca-012a-4efa-a43b-2a11844f241e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.454272 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.454228 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f9d65ca-012a-4efa-a43b-2a11844f241e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.454327 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.454310 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9d65ca-012a-4efa-a43b-2a11844f241e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.454503 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.454477 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f9d65ca-012a-4efa-a43b-2a11844f241e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.456580 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.456557 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f9d65ca-012a-4efa-a43b-2a11844f241e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.456877 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.456854 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f9d65ca-012a-4efa-a43b-2a11844f241e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.474438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.474411 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2np\" (UniqueName: \"kubernetes.io/projected/5f9d65ca-012a-4efa-a43b-2a11844f241e-kube-api-access-nv2np\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9\" (UID: \"5f9d65ca-012a-4efa-a43b-2a11844f241e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.493785 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.493757 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:02.602413 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.602373 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" event={"ID":"d3d2a960-6069-4a49-a738-e07f9178d7a0","Type":"ContainerStarted","Data":"b23896b49ee33bf5f197298699c1ae670a4f7c8f53324ef01e561e8f329152e6"} Nov 03 14:20:02.802385 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.802303 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9"] Nov 03 14:20:02.805491 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:20:02.805463 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9d65ca_012a_4efa_a43b_2a11844f241e.slice/crio-a2ac79c715e78a41ea1c9f3b5e5698e3fa4b667e20b4e159dd1b0b1efbe54bed WatchSource:0}: Error finding container a2ac79c715e78a41ea1c9f3b5e5698e3fa4b667e20b4e159dd1b0b1efbe54bed: Status 404 returned error can't find the container with id a2ac79c715e78a41ea1c9f3b5e5698e3fa4b667e20b4e159dd1b0b1efbe54bed Nov 03 14:20:02.987078 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:02.987047 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:20:03.001932 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:03.001906 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:20:03.609349 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:03.609313 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" event={"ID":"5f9d65ca-012a-4efa-a43b-2a11844f241e","Type":"ContainerStarted","Data":"77ccf07da0d1cc2713f088048f3edc4a399437958ca6d3f8e15fdc38e9732351"} Nov 03 14:20:03.609905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:03.609372 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" event={"ID":"5f9d65ca-012a-4efa-a43b-2a11844f241e","Type":"ContainerStarted","Data":"a2ac79c715e78a41ea1c9f3b5e5698e3fa4b667e20b4e159dd1b0b1efbe54bed"} Nov 03 14:20:03.624041 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:03.624014 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4" Nov 03 14:20:03.650269 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:03.650226 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" podStartSLOduration=1.046136462 podStartE2EDuration="1.650210986s" totalImagesPullingTime="604.074524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:03.649726543 +0000 UTC m=+479.208363027" watchObservedRunningTime="2026-11-03 14:20:03.650210986 +0000 UTC m=+479.208847466" Nov 03 14:20:04.494164 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:04.494134 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:04.616466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:04.616432 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" event={"ID":"d3d2a960-6069-4a49-a738-e07f9178d7a0","Type":"ContainerStarted","Data":"4627d18a515ea73ef550dc5a6cc34f37568ffc1595ffc69f501fd56bc8bee4e8"} Nov 03 14:20:04.657141 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:04.657096 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" podStartSLOduration=1.75020523 podStartE2EDuration="3.657083043s" totalImagesPullingTime="1.906877813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:04.656582039 +0000 UTC m=+480.215218524" watchObservedRunningTime="2026-11-03 14:20:04.657083043 +0000 UTC m=+480.215719505" Nov 03 14:20:05.478468 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:05.478434 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:06.095920 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:06.095881 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:20:06.108601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:06.108572 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:20:06.635551 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:06.635527 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d77tr" Nov 03 14:20:09.962205 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:09.962175 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:20:09.974736 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:09.974713 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:20:10.658256 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:10.658229 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n" Nov 03 14:20:11.478159 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:11.478122 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:11.490551 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:11.490528 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:11.662597 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:11.662571 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-c242s" Nov 03 14:20:12.494477 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:12.494447 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:12.507180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:12.507159 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:12.667242 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:12.667206 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9" Nov 03 14:20:18.406900 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.406863 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7"] Nov 03 14:20:18.410636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.410598 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.427755 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.427730 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4nspm\"" Nov 03 14:20:18.440073 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.440046 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7"] Nov 03 14:20:18.490165 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.490132 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.490165 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.490163 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4465\" (UniqueName: \"kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-kube-api-access-h4465\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.490363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.490186 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.490363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.490253 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.490363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.490326 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.490363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.490347 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.490490 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.490419 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.490490 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.490450 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.490490 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.490470 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.591099 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.591066 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.591099 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.591096 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.591331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.591121 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.591331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.591245 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.591331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.591281 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.591480 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.591339 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.591480 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.591366 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4465\" (UniqueName: \"kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-kube-api-access-h4465\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.591480 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.591390 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.591480 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.591414 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.591723 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.591581 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.598988 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.592139 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.598988 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.592475 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.598988 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.592496 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.598988 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.592520 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.598988 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.594661 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.599397 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.599374 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.613928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.613904 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.614115 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.614096 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4465\" (UniqueName: \"kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-kube-api-access-h4465\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-92rg7\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:18.723307 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:18.723276 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:19.035457 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:19.035434 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7"] Nov 03 14:20:19.037250 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:20:19.037228 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4314dc_1418_4cf6_ba99_4e3f448e3cab.slice/crio-6b197985312316143855e5e415a0ace4b975068c70b599b6a85bb02ccb8a15be WatchSource:0}: Error finding container 6b197985312316143855e5e415a0ace4b975068c70b599b6a85bb02ccb8a15be: Status 404 returned error can't find the container with id 6b197985312316143855e5e415a0ace4b975068c70b599b6a85bb02ccb8a15be Nov 03 14:20:19.039306 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:19.039286 6389 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:20:19.688361 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:19.688328 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" event={"ID":"2e4314dc-1418-4cf6-ba99-4e3f448e3cab","Type":"ContainerStarted","Data":"6b197985312316143855e5e415a0ace4b975068c70b599b6a85bb02ccb8a15be"} Nov 03 14:20:21.709059 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:21.709027 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:20:21.709330 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:21.709097 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:20:21.709330 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:21.709124 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:20:22.704121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:22.704080 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" event={"ID":"2e4314dc-1418-4cf6-ba99-4e3f448e3cab","Type":"ContainerStarted","Data":"9d451ec677fba2760dfaca2e36afbd477810c18918139944c51748ec98ff41da"} Nov 03 14:20:22.737632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:22.737221 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:22.743788 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:22.743751 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-92rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 14:20:22.743962 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:22.743812 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 14:20:22.788976 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:22.788928 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" podStartSLOduration=2.12036066 podStartE2EDuration="4.788912905s" totalImagesPullingTime="2.668552245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:22.776462795 +0000 UTC m=+498.335099279" watchObservedRunningTime="2026-11-03 14:20:22.788912905 +0000 UTC m=+498.347549388" Nov 03 14:20:23.737634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:23.733044 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-92rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 14:20:23.737634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:23.733109 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 14:20:24.742633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:24.742019 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-92rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 14:20:24.742633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:24.742096 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 14:20:25.739630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:25.738975 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-92rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 14:20:25.739630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:25.739048 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 14:20:26.733645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:26.732977 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-92rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 14:20:26.733645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:26.733055 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 14:20:27.724139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:27.724096 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-92rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 14:20:27.724331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:27.724168 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 14:20:28.373052 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:20:28.373017 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:20:28.723558 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:28.723525 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-92rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 14:20:28.723762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:28.723583 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 14:20:29.723726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:29.723690 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-92rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 14:20:29.724121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:29.723769 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 14:20:30.886985 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:30.886941 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:30.887461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:30.887354 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:30.888229 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:30.888205 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:20:48.520066 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.520027 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28"] Nov 03 14:20:48.524113 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.524091 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.553905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.553874 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28"] Nov 03 14:20:48.559453 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.559428 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs"] Nov 03 14:20:48.562925 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.562905 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.585750 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.585727 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs"] Nov 03 14:20:48.688252 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688218 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9dsf\" (UniqueName: \"kubernetes.io/projected/e5733a68-00fa-4304-87d1-e10764bf1d6c-kube-api-access-f9dsf\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.688252 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688253 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.688454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688276 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.688454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688356 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.688454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688416 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f9066-8666-4172-bf60-4ff8759555bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.688454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688450 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.688582 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688473 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.688582 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688511 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.688582 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688551 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.688582 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688572 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.688727 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688636 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.688727 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688680 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5733a68-00fa-4304-87d1-e10764bf1d6c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.688791 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688731 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5733a68-00fa-4304-87d1-e10764bf1d6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.688791 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688748 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5733a68-00fa-4304-87d1-e10764bf1d6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.688851 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688800 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw5hb\" (UniqueName: \"kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-kube-api-access-xw5hb\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.688851 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688818 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.688851 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688839 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.688941 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.688857 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.789741 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.789649 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.789741 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.789693 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.789981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.789751 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9dsf\" (UniqueName: \"kubernetes.io/projected/e5733a68-00fa-4304-87d1-e10764bf1d6c-kube-api-access-f9dsf\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.789981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.789776 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.789981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.789807 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.789981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.789836 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.789981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.789870 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f9066-8666-4172-bf60-4ff8759555bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.789981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.789899 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.789981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.789927 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.789981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.789961 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.790408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.789989 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.790408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790021 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.790408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790049 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.790408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790062 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.790408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790095 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5733a68-00fa-4304-87d1-e10764bf1d6c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.790408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790125 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.790408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790139 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5733a68-00fa-4304-87d1-e10764bf1d6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.790408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790184 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5733a68-00fa-4304-87d1-e10764bf1d6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.790408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790239 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw5hb\" (UniqueName: \"kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-kube-api-access-xw5hb\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.790408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790266 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.790408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790323 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.791031 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790547 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.791031 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790652 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f9066-8666-4172-bf60-4ff8759555bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.791031 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790686 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.791031 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790762 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.791031 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.790905 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5733a68-00fa-4304-87d1-e10764bf1d6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.791581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.791556 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.791752 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.791582 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.792830 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.792807 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5733a68-00fa-4304-87d1-e10764bf1d6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.793026 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.793009 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5733a68-00fa-4304-87d1-e10764bf1d6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.793266 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.793243 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.793344 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.793290 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.810901 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.810871 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9dsf\" (UniqueName: \"kubernetes.io/projected/e5733a68-00fa-4304-87d1-e10764bf1d6c-kube-api-access-f9dsf\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.811110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.811090 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw5hb\" (UniqueName: \"kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-kube-api-access-xw5hb\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.811178 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.811110 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z7p28\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.811243 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.811215 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5733a68-00fa-4304-87d1-e10764bf1d6c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7jwxs\" (UID: \"e5733a68-00fa-4304-87d1-e10764bf1d6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:48.837402 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.837368 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:48.878489 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:48.878463 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:49.154683 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.154659 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28"] Nov 03 14:20:49.156371 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:20:49.156344 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9f9066_8666_4172_bf60_4ff8759555bb.slice/crio-92e74b2972378dc092b63e75f8d790560283fc1658e9a3769f4b98d9b5611290 WatchSource:0}: Error finding container 92e74b2972378dc092b63e75f8d790560283fc1658e9a3769f4b98d9b5611290: Status 404 returned error can't find the container with id 92e74b2972378dc092b63e75f8d790560283fc1658e9a3769f4b98d9b5611290 Nov 03 14:20:49.158424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.158393 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:20:49.158496 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.158455 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:20:49.158496 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.158490 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:20:49.194054 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.193719 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs"] Nov 03 14:20:49.196850 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:20:49.196818 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5733a68_00fa_4304_87d1_e10764bf1d6c.slice/crio-8500d9a8a033d638109b46cf6774e147796f01ee598945379dc0a18d6185aec4 WatchSource:0}: Error finding container 8500d9a8a033d638109b46cf6774e147796f01ee598945379dc0a18d6185aec4: Status 404 returned error can't find the container with id 8500d9a8a033d638109b46cf6774e147796f01ee598945379dc0a18d6185aec4 Nov 03 14:20:49.199128 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.199096 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:20:49.199213 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.199163 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:20:49.199213 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.199193 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:20:49.868698 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.868645 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" event={"ID":"8d9f9066-8666-4172-bf60-4ff8759555bb","Type":"ContainerStarted","Data":"9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1"} Nov 03 14:20:49.869171 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.868720 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" event={"ID":"8d9f9066-8666-4172-bf60-4ff8759555bb","Type":"ContainerStarted","Data":"92e74b2972378dc092b63e75f8d790560283fc1658e9a3769f4b98d9b5611290"} Nov 03 14:20:49.870658 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.870632 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" event={"ID":"e5733a68-00fa-4304-87d1-e10764bf1d6c","Type":"ContainerStarted","Data":"acaec202e1d8f2796e8c837b9a9ffe0d1da09146e3ceb302725f2fa794ac5537"} Nov 03 14:20:49.870843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.870819 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" event={"ID":"e5733a68-00fa-4304-87d1-e10764bf1d6c","Type":"ContainerStarted","Data":"8500d9a8a033d638109b46cf6774e147796f01ee598945379dc0a18d6185aec4"} Nov 03 14:20:49.879094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.879074 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:49.986579 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.986513 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podStartSLOduration=1.986493838 podStartE2EDuration="1.986493838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:49.937920373 +0000 UTC m=+525.496556863" watchObservedRunningTime="2026-11-03 14:20:49.986493838 +0000 UTC m=+525.545130331" Nov 03 14:20:49.988166 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:49.988106 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podStartSLOduration=1.9880945410000002 podStartE2EDuration="1.988094541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:49.985500331 +0000 UTC m=+525.544136817" watchObservedRunningTime="2026-11-03 14:20:49.988094541 +0000 UTC m=+525.546731020" Nov 03 14:20:50.840634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:50.839987 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:50.844629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:50.843810 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 14:20:50.844629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:50.843873 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 14:20:50.880958 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:50.880721 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 14:20:50.880958 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:50.880770 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 14:20:51.841097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:51.841051 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 14:20:51.841404 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:51.841373 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 14:20:51.880055 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:51.880020 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 14:20:51.880251 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:51.880103 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 14:20:52.842644 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:52.839683 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 14:20:52.842644 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:52.839747 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 14:20:52.879097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:52.879059 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 14:20:52.879272 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:52.879110 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 14:20:53.839055 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:53.839016 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 14:20:53.839381 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:53.839347 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 14:20:53.879277 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:53.879245 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 14:20:53.879703 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:53.879311 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 14:20:54.844634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:54.844007 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 14:20:54.844634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:54.844073 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 14:20:54.878877 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:54.878760 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 14:20:54.878877 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:54.878813 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 14:20:55.367486 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.367440 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g"] Nov 03 14:20:55.384091 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.383853 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.396905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.396875 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g"] Nov 03 14:20:55.404331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.404299 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4429aa-openshift-default-dockercfg-rph6j\"" Nov 03 14:20:55.497943 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.497901 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-istio-token\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.498133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.497959 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-credential-socket\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.498133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.498117 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-data\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.498218 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.498159 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d691739-5007-41c9-acf9-445e68d9561f-istiod-ca-cert\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.498278 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.498254 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-socket\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.498408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.498390 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2wk2\" (UniqueName: \"kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-kube-api-access-p2wk2\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.498445 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.498425 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-certs\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.498494 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.498457 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-envoy\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.498566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.498539 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d691739-5007-41c9-acf9-445e68d9561f-istio-podinfo\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.600012 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.599976 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2wk2\" (UniqueName: \"kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-kube-api-access-p2wk2\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.600012 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.600015 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-certs\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.600272 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.600034 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-envoy\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.600272 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.600052 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d691739-5007-41c9-acf9-445e68d9561f-istio-podinfo\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.600272 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.600208 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-istio-token\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.600272 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.600242 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-credential-socket\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.600488 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.600307 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-data\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.600488 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.600333 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d691739-5007-41c9-acf9-445e68d9561f-istiod-ca-cert\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.600488 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.600367 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-socket\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.600672 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.600498 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-certs\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.600728 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.600685 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-credential-socket\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.601029 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.601005 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-socket\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.601282 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.601264 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-data\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.601343 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.601284 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d691739-5007-41c9-acf9-445e68d9561f-istiod-ca-cert\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.602941 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.602910 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-envoy\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.603506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.603484 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d691739-5007-41c9-acf9-445e68d9561f-istio-podinfo\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.621849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.621791 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2wk2\" (UniqueName: \"kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-kube-api-access-p2wk2\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.621933 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.621903 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-istio-token\") pod \"e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.705855 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.705818 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:55.838509 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.838478 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 14:20:55.838705 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.838532 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 14:20:55.879321 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.879227 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 14:20:55.879321 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:55.879280 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 14:20:56.035676 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:56.035643 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g"] Nov 03 14:20:56.039709 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:20:56.039671 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d691739_5007_41c9_acf9_445e68d9561f.slice/crio-fa3a7fdcf882dde5077215d687830fba3868451161bb6c76afcf341e9eebb349 WatchSource:0}: Error finding container fa3a7fdcf882dde5077215d687830fba3868451161bb6c76afcf341e9eebb349: Status 404 returned error can't find the container with id fa3a7fdcf882dde5077215d687830fba3868451161bb6c76afcf341e9eebb349 Nov 03 14:20:56.041966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:56.041932 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:20:56.042086 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:56.041999 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:20:56.042086 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:56.042028 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:20:56.838197 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:56.838163 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 14:20:56.838759 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:56.838221 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 14:20:56.879841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:56.879805 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 14:20:56.880047 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:56.879872 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 14:20:56.911869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:56.911831 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" event={"ID":"4d691739-5007-41c9-acf9-445e68d9561f","Type":"ContainerStarted","Data":"82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce"} Nov 03 14:20:56.912080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:56.911897 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" event={"ID":"4d691739-5007-41c9-acf9-445e68d9561f","Type":"ContainerStarted","Data":"fa3a7fdcf882dde5077215d687830fba3868451161bb6c76afcf341e9eebb349"} Nov 03 14:20:56.956447 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:56.956391 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" podStartSLOduration=1.956378575 podStartE2EDuration="1.956378575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:56.956078403 +0000 UTC m=+532.514714887" watchObservedRunningTime="2026-11-03 14:20:56.956378575 +0000 UTC m=+532.515015059" Nov 03 14:20:57.706749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:57.706663 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:57.714471 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:57.714441 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:57.920651 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:57.920136 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:57.921171 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:57.921143 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:20:58.387239 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:20:58.387103 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:20:58.776110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:58.772545 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:58.776110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:58.774245 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:58.776110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:58.775892 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 14:20:58.880776 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:58.880741 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:20:58.880967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:58.880799 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:20:58.904025 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:58.903998 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:58.904456 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:58.904396 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:20:58.906161 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:20:58.906125 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" Nov 03 14:21:00.130139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.130103 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz"] Nov 03 14:21:00.133961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.133937 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" Nov 03 14:21:00.161908 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.161876 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-156ae1-dockercfg-6tv6h\"" Nov 03 14:21:00.161908 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.161884 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-156ae1\"" Nov 03 14:21:00.163554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.163528 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz"] Nov 03 14:21:00.164892 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.164871 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd"] Nov 03 14:21:00.168448 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.168429 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.185463 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.185436 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd"] Nov 03 14:21:00.251285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.251251 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmq9\" (UniqueName: \"kubernetes.io/projected/1739d708-b46f-4045-862e-967f64afd54b-kube-api-access-qqmq9\") pod \"payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz\" (UID: \"1739d708-b46f-4045-862e-967f64afd54b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" Nov 03 14:21:00.251477 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.251304 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1739d708-b46f-4045-862e-967f64afd54b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz\" (UID: \"1739d708-b46f-4045-862e-967f64afd54b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" Nov 03 14:21:00.251477 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.251362 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-156ae1-serving-cert\"" Nov 03 14:21:00.359934 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.359895 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58d70cb-6a02-44ac-b026-da52f2b1ad12-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.360139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.359943 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-metrics-certs\") pod \"maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.360139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.359998 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42cv9\" (UniqueName: \"kubernetes.io/projected/c58d70cb-6a02-44ac-b026-da52f2b1ad12-kube-api-access-42cv9\") pod \"maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.360139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.360039 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmq9\" (UniqueName: \"kubernetes.io/projected/1739d708-b46f-4045-862e-967f64afd54b-kube-api-access-qqmq9\") pod \"payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz\" (UID: \"1739d708-b46f-4045-862e-967f64afd54b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" Nov 03 14:21:00.360139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.360095 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1739d708-b46f-4045-862e-967f64afd54b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz\" (UID: \"1739d708-b46f-4045-862e-967f64afd54b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" Nov 03 14:21:00.360411 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.360152 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-maas-api-tls\") pod \"maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.362928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.362896 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1739d708-b46f-4045-862e-967f64afd54b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz\" (UID: \"1739d708-b46f-4045-862e-967f64afd54b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" Nov 03 14:21:00.383562 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.383491 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmq9\" (UniqueName: \"kubernetes.io/projected/1739d708-b46f-4045-862e-967f64afd54b-kube-api-access-qqmq9\") pod \"payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz\" (UID: \"1739d708-b46f-4045-862e-967f64afd54b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" Nov 03 14:21:00.448260 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.448225 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" Nov 03 14:21:00.461728 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.461702 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-maas-api-tls\") pod \"maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.461843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.461801 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58d70cb-6a02-44ac-b026-da52f2b1ad12-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.461843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.461829 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-metrics-certs\") pod \"maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.461956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.461872 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42cv9\" (UniqueName: \"kubernetes.io/projected/c58d70cb-6a02-44ac-b026-da52f2b1ad12-kube-api-access-42cv9\") pod \"maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.462633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.462559 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58d70cb-6a02-44ac-b026-da52f2b1ad12-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.464559 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.464538 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-metrics-certs\") pod \"maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.464686 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.464674 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-maas-api-tls\") pod \"maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.482078 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.482048 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42cv9\" (UniqueName: \"kubernetes.io/projected/c58d70cb-6a02-44ac-b026-da52f2b1ad12-kube-api-access-42cv9\") pod \"maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.487762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.487739 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:00.753803 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.753766 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz"] Nov 03 14:21:00.757051 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:21:00.757021 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1739d708_b46f_4045_862e_967f64afd54b.slice/crio-e0e0cf4dee14d74ab970a6568162f717dba9a1557949a5d0290e82c2562828fe WatchSource:0}: Error finding container e0e0cf4dee14d74ab970a6568162f717dba9a1557949a5d0290e82c2562828fe: Status 404 returned error can't find the container with id e0e0cf4dee14d74ab970a6568162f717dba9a1557949a5d0290e82c2562828fe Nov 03 14:21:00.802082 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.801938 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd"] Nov 03 14:21:00.804596 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:21:00.804567 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58d70cb_6a02_44ac_b026_da52f2b1ad12.slice/crio-2ed15fa561931023bb38996dee4ab07be573a1e08d52615d0dfa6bc1c9c7b4bc WatchSource:0}: Error finding container 2ed15fa561931023bb38996dee4ab07be573a1e08d52615d0dfa6bc1c9c7b4bc: Status 404 returned error can't find the container with id 2ed15fa561931023bb38996dee4ab07be573a1e08d52615d0dfa6bc1c9c7b4bc Nov 03 14:21:00.936018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.935979 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" event={"ID":"1739d708-b46f-4045-862e-967f64afd54b","Type":"ContainerStarted","Data":"b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b"} Nov 03 14:21:00.936198 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.936049 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" event={"ID":"1739d708-b46f-4045-862e-967f64afd54b","Type":"ContainerStarted","Data":"e0e0cf4dee14d74ab970a6568162f717dba9a1557949a5d0290e82c2562828fe"} Nov 03 14:21:00.936198 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.936188 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" Nov 03 14:21:00.937420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.937392 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" event={"ID":"c58d70cb-6a02-44ac-b026-da52f2b1ad12","Type":"ContainerStarted","Data":"2ed15fa561931023bb38996dee4ab07be573a1e08d52615d0dfa6bc1c9c7b4bc"} Nov 03 14:21:00.986137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:00.986083 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" podStartSLOduration=0.986070967 podStartE2EDuration="986.070967ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:00.98540196 +0000 UTC m=+536.544038445" watchObservedRunningTime="2026-11-03 14:21:00.986070967 +0000 UTC m=+536.544707451" Nov 03 14:21:01.943705 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:01.943655 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" event={"ID":"c58d70cb-6a02-44ac-b026-da52f2b1ad12","Type":"ContainerStarted","Data":"14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7"} Nov 03 14:21:01.944152 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:01.943931 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:01.984893 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:01.984831 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" podStartSLOduration=1.563326144 podStartE2EDuration="1.984816312s" totalImagesPullingTime="421.490168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:01.983577162 +0000 UTC m=+537.542213647" watchObservedRunningTime="2026-11-03 14:21:01.984816312 +0000 UTC m=+537.543452799" Nov 03 14:21:02.465516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.465476 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl"] Nov 03 14:21:02.468866 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.468843 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:02.480872 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.480846 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:02.481010 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.480921 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vf7c\" (UniqueName: \"kubernetes.io/projected/0952ddc3-f129-4d1e-8d43-c73e01cf8258-kube-api-access-4vf7c\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:02.481010 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.480945 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0952ddc3-f129-4d1e-8d43-c73e01cf8258-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:02.481101 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.481082 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-metrics-certs\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:02.486020 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.486002 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-4429aa-serving-cert\"" Nov 03 14:21:02.499681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.499652 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl"] Nov 03 14:21:02.582087 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.582053 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vf7c\" (UniqueName: \"kubernetes.io/projected/0952ddc3-f129-4d1e-8d43-c73e01cf8258-kube-api-access-4vf7c\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:02.582240 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.582117 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0952ddc3-f129-4d1e-8d43-c73e01cf8258-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:02.582240 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.582179 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-metrics-certs\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:02.582368 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.582243 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:02.582368 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:21:02.582349 6389 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-serving-cert: secret "maas-api-e2e-praxis-a-4429aa-serving-cert" not found Nov 03 14:21:02.582506 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:21:02.582461 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-maas-api-tls podName:0952ddc3-f129-4d1e-8d43-c73e01cf8258 nodeName:}" failed. No retries permitted until 2026-11-03 14:21:03.082434483 +0000 UTC m=+538.641070965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-maas-api-tls") pod "maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" (UID: "0952ddc3-f129-4d1e-8d43-c73e01cf8258") : secret "maas-api-e2e-praxis-a-4429aa-serving-cert" not found Nov 03 14:21:02.582970 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.582940 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0952ddc3-f129-4d1e-8d43-c73e01cf8258-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:02.585119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.585088 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-metrics-certs\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:02.607438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:02.607418 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vf7c\" (UniqueName: \"kubernetes.io/projected/0952ddc3-f129-4d1e-8d43-c73e01cf8258-kube-api-access-4vf7c\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:03.086469 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:03.086417 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:03.091167 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:03.091133 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:03.382865 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:03.382778 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:03.684659 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:03.684636 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl"] Nov 03 14:21:03.686376 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:21:03.686345 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0952ddc3_f129_4d1e_8d43_c73e01cf8258.slice/crio-00b2d5710f5003d5d8dc86ce42906b1dbef88549c2ae82297492018b40068d63 WatchSource:0}: Error finding container 00b2d5710f5003d5d8dc86ce42906b1dbef88549c2ae82297492018b40068d63: Status 404 returned error can't find the container with id 00b2d5710f5003d5d8dc86ce42906b1dbef88549c2ae82297492018b40068d63 Nov 03 14:21:03.954868 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:03.954782 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" event={"ID":"0952ddc3-f129-4d1e-8d43-c73e01cf8258","Type":"ContainerStarted","Data":"00b2d5710f5003d5d8dc86ce42906b1dbef88549c2ae82297492018b40068d63"} Nov 03 14:21:04.665509 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.665480 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg"] Nov 03 14:21:04.669956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.669934 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:04.685725 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.685703 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-86c2e2-serving-cert\"" Nov 03 14:21:04.698843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.698787 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg"] Nov 03 14:21:04.707391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.707353 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:04.707528 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.707462 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-metrics-certs\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:04.707584 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.707560 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bh5x\" (UniqueName: \"kubernetes.io/projected/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-kube-api-access-2bh5x\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:04.707725 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.707702 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:04.809012 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.808983 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bh5x\" (UniqueName: \"kubernetes.io/projected/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-kube-api-access-2bh5x\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:04.809232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.809027 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:04.809232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.809096 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:04.809232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.809116 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-metrics-certs\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:04.809232 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:21:04.809221 6389 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-serving-cert: secret "maas-api-e2e-worker-w0-86c2e2-serving-cert" not found Nov 03 14:21:04.809471 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:21:04.809298 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-maas-api-tls podName:3c39da63-41cb-41ac-b65a-a4f51ffc9ba2 nodeName:}" failed. No retries permitted until 2026-11-03 14:21:05.309277415 +0000 UTC m=+540.867913878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-maas-api-tls") pod "maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" (UID: "3c39da63-41cb-41ac-b65a-a4f51ffc9ba2") : secret "maas-api-e2e-worker-w0-86c2e2-serving-cert" not found Nov 03 14:21:04.809683 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.809665 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:04.811653 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.811636 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-metrics-certs\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:04.829278 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.829255 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bh5x\" (UniqueName: \"kubernetes.io/projected/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-kube-api-access-2bh5x\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:04.961906 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.961862 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" event={"ID":"0952ddc3-f129-4d1e-8d43-c73e01cf8258","Type":"ContainerStarted","Data":"8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3"} Nov 03 14:21:04.962106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:04.962025 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:05.005272 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:05.005219 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" podStartSLOduration=2.666564134 podStartE2EDuration="3.005206239s" totalImagesPullingTime="338.642105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:05.003358171 +0000 UTC m=+540.561994655" watchObservedRunningTime="2026-11-03 14:21:05.005206239 +0000 UTC m=+540.563842723" Nov 03 14:21:05.313558 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:05.313480 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:05.316042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:05.316022 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:05.583207 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:05.583118 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:05.894843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:05.894815 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg"] Nov 03 14:21:05.896514 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:21:05.896482 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c39da63_41cb_41ac_b65a_a4f51ffc9ba2.slice/crio-8ba6579505af409fd966786b1b283b8f3d857942ca34d5ba729d4d2d539c96b9 WatchSource:0}: Error finding container 8ba6579505af409fd966786b1b283b8f3d857942ca34d5ba729d4d2d539c96b9: Status 404 returned error can't find the container with id 8ba6579505af409fd966786b1b283b8f3d857942ca34d5ba729d4d2d539c96b9 Nov 03 14:21:05.972462 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:05.972424 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" event={"ID":"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2","Type":"ContainerStarted","Data":"8ba6579505af409fd966786b1b283b8f3d857942ca34d5ba729d4d2d539c96b9"} Nov 03 14:21:06.978638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:06.978586 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" event={"ID":"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2","Type":"ContainerStarted","Data":"172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b"} Nov 03 14:21:06.979027 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:06.978756 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:07.020482 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:07.020426 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" podStartSLOduration=2.719646053 podStartE2EDuration="3.020411292s" totalImagesPullingTime="300.765239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:07.018845523 +0000 UTC m=+542.577482006" watchObservedRunningTime="2026-11-03 14:21:07.020411292 +0000 UTC m=+542.579047776" Nov 03 14:21:07.953649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:07.953602 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:21:10.978952 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:10.978924 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:21:11.946543 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:11.946515 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" Nov 03 14:21:12.991434 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:12.991404 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:21:18.037019 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:18.036983 6389 generic.go:356] "Generic (PLEG): container finished" podID="4d691739-5007-41c9-acf9-445e68d9561f" containerID="82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce" exitCode=0 Nov 03 14:21:18.037634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:18.037059 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" event={"ID":"4d691739-5007-41c9-acf9-445e68d9561f","Type":"ContainerDied","Data":"82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce"} Nov 03 14:21:18.037634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:18.037450 6389 scope.go:122] "RemoveContainer" containerID="82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce" Nov 03 14:21:18.038687 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:18.038660 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:21:18.038739 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:18.038726 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:21:18.038773 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:18.038761 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:21:19.067010 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:19.066973 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" event={"ID":"4d691739-5007-41c9-acf9-445e68d9561f","Type":"ContainerStarted","Data":"b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755"} Nov 03 14:21:19.706634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:19.706345 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:21:19.706634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:19.706439 6389 patch_prober.go:29] interesting pod/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 14:21:19.706634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:19.706482 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" podUID="4d691739-5007-41c9-acf9-445e68d9561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 14:21:20.714635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:20.709168 6389 patch_prober.go:29] interesting pod/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 14:21:20.714635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:20.709225 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" podUID="4d691739-5007-41c9-acf9-445e68d9561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 14:21:21.713640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:21.713173 6389 patch_prober.go:29] interesting pod/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 14:21:21.713640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:21.713237 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" podUID="4d691739-5007-41c9-acf9-445e68d9561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 14:21:22.706382 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:22.706343 6389 patch_prober.go:29] interesting pod/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 14:21:22.706931 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:22.706397 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" podUID="4d691739-5007-41c9-acf9-445e68d9561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 14:21:23.707132 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:23.707095 6389 patch_prober.go:29] interesting pod/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 14:21:23.707559 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:23.707156 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" podUID="4d691739-5007-41c9-acf9-445e68d9561f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 14:21:24.711093 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:24.711055 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:21:24.711565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:24.711478 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:21:24.712063 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:24.712044 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:21:26.330214 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.330177 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59"] Nov 03 14:21:26.333941 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.333919 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.354498 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.354472 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59"] Nov 03 14:21:26.365133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.365106 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-86c2e2\"/\"openshift-service-ca.crt\"" Nov 03 14:21:26.365509 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.365487 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-86c2e2\"/\"simulator-w0-86c2e2-kserve-self-signed-certs\"" Nov 03 14:21:26.365647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.365518 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-86c2e2\"/\"kube-root-ca.crt\"" Nov 03 14:21:26.368048 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.368035 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-86c2e2\"/\"default-dockercfg-xchvf\"" Nov 03 14:21:26.481303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.481268 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tls-certs\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.481303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.481306 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-home\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.481513 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.481329 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-model-cache\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.481513 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.481454 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-dshm\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.481577 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.481552 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7xvt\" (UniqueName: \"kubernetes.io/projected/76e983d7-ca69-44d1-857c-65eb03ec5e6a-kube-api-access-w7xvt\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.481647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.481588 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tmp-dir\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.583275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.583174 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xvt\" (UniqueName: \"kubernetes.io/projected/76e983d7-ca69-44d1-857c-65eb03ec5e6a-kube-api-access-w7xvt\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.583275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.583239 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tmp-dir\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.583511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.583313 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tls-certs\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.583511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.583428 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-home\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.583511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.583467 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-model-cache\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.583511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.583507 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-dshm\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.583764 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.583715 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tmp-dir\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.583818 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.583805 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-home\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.583872 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.583845 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-model-cache\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.585876 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.585853 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-dshm\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.586022 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.586004 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tls-certs\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.605138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.605110 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xvt\" (UniqueName: \"kubernetes.io/projected/76e983d7-ca69-44d1-857c-65eb03ec5e6a-kube-api-access-w7xvt\") pod \"simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:26.644890 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:26.644857 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:27.007474 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:27.007450 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59"] Nov 03 14:21:27.008776 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:21:27.008749 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76e983d7_ca69_44d1_857c_65eb03ec5e6a.slice/crio-0a29670979c565aca271e68b89fb71503107b2d212edbf0e1f0eaa6bb056804e WatchSource:0}: Error finding container 0a29670979c565aca271e68b89fb71503107b2d212edbf0e1f0eaa6bb056804e: Status 404 returned error can't find the container with id 0a29670979c565aca271e68b89fb71503107b2d212edbf0e1f0eaa6bb056804e Nov 03 14:21:27.165538 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:27.165504 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" event={"ID":"76e983d7-ca69-44d1-857c-65eb03ec5e6a","Type":"ContainerStarted","Data":"78208333ea53d9195e7016e168d7d3f10353db9d028157238d9c725f50c17519"} Nov 03 14:21:27.165538 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:27.165544 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" event={"ID":"76e983d7-ca69-44d1-857c-65eb03ec5e6a","Type":"ContainerStarted","Data":"0a29670979c565aca271e68b89fb71503107b2d212edbf0e1f0eaa6bb056804e"} Nov 03 14:21:27.227126 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:27.227074 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" podStartSLOduration=1.227060058 podStartE2EDuration="1.227060058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:27.226411971 +0000 UTC m=+562.785048456" watchObservedRunningTime="2026-11-03 14:21:27.227060058 +0000 UTC m=+562.785696541" Nov 03 14:21:27.644954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:27.644925 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:28.343393 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.343354 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm"] Nov 03 14:21:28.346042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.346021 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" Nov 03 14:21:28.374431 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.374396 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-0a0ec6\"" Nov 03 14:21:28.374652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.374519 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-0a0ec6-dockercfg-brnqg\"" Nov 03 14:21:28.377749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.377724 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm"] Nov 03 14:21:28.379874 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:21:28.379841 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:21:28.503394 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.503361 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndmc4\" (UniqueName: \"kubernetes.io/projected/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-kube-api-access-ndmc4\") pod \"payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm\" (UID: \"cd7d2837-bbab-4c2a-acdc-f259a2e80e30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" Nov 03 14:21:28.503563 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.503448 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm\" (UID: \"cd7d2837-bbab-4c2a-acdc-f259a2e80e30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" Nov 03 14:21:28.605038 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.604958 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndmc4\" (UniqueName: \"kubernetes.io/projected/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-kube-api-access-ndmc4\") pod \"payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm\" (UID: \"cd7d2837-bbab-4c2a-acdc-f259a2e80e30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" Nov 03 14:21:28.605211 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.605063 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm\" (UID: \"cd7d2837-bbab-4c2a-acdc-f259a2e80e30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" Nov 03 14:21:28.605834 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.605809 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm\" (UID: \"cd7d2837-bbab-4c2a-acdc-f259a2e80e30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" Nov 03 14:21:28.627341 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.627306 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndmc4\" (UniqueName: \"kubernetes.io/projected/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-kube-api-access-ndmc4\") pod \"payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm\" (UID: \"cd7d2837-bbab-4c2a-acdc-f259a2e80e30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" Nov 03 14:21:28.658977 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.658950 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" Nov 03 14:21:28.968583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:28.968555 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm"] Nov 03 14:21:28.970186 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:21:28.970151 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7d2837_bbab_4c2a_acdc_f259a2e80e30.slice/crio-a76c93b65743793daa25bafdd24bdad8502b8bee9440271121ba6be047a6cd8f WatchSource:0}: Error finding container a76c93b65743793daa25bafdd24bdad8502b8bee9440271121ba6be047a6cd8f: Status 404 returned error can't find the container with id a76c93b65743793daa25bafdd24bdad8502b8bee9440271121ba6be047a6cd8f Nov 03 14:21:29.178383 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:29.178343 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" event={"ID":"cd7d2837-bbab-4c2a-acdc-f259a2e80e30","Type":"ContainerStarted","Data":"3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382"} Nov 03 14:21:29.178595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:29.178416 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" event={"ID":"cd7d2837-bbab-4c2a-acdc-f259a2e80e30","Type":"ContainerStarted","Data":"a76c93b65743793daa25bafdd24bdad8502b8bee9440271121ba6be047a6cd8f"} Nov 03 14:21:29.178815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:29.178799 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" Nov 03 14:21:29.245395 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:29.245291 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" podStartSLOduration=1.245276144 podStartE2EDuration="1.245276144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:29.243286535 +0000 UTC m=+564.801923019" watchObservedRunningTime="2026-11-03 14:21:29.245276144 +0000 UTC m=+564.803912629" Nov 03 14:21:34.601785 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.601751 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65"] Nov 03 14:21:34.604843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.604822 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.628001 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.627972 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-0a0ec6-serving-cert\"" Nov 03 14:21:34.640520 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.640488 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65"] Nov 03 14:21:34.782083 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.782042 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-metrics-certs\") pod \"maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.782271 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.782223 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cdkd\" (UniqueName: \"kubernetes.io/projected/846a80aa-400e-4062-a2b5-057ea0124554-kube-api-access-7cdkd\") pod \"maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.782350 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.782326 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.782544 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.782520 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846a80aa-400e-4062-a2b5-057ea0124554-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.883804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.883712 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cdkd\" (UniqueName: \"kubernetes.io/projected/846a80aa-400e-4062-a2b5-057ea0124554-kube-api-access-7cdkd\") pod \"maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.883804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.883756 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.884046 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.883872 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846a80aa-400e-4062-a2b5-057ea0124554-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.884046 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.883951 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-metrics-certs\") pod \"maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.884549 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.884521 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846a80aa-400e-4062-a2b5-057ea0124554-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.886480 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.886456 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-metrics-certs\") pod \"maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.886813 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.886789 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.906596 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.906570 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cdkd\" (UniqueName: \"kubernetes.io/projected/846a80aa-400e-4062-a2b5-057ea0124554-kube-api-access-7cdkd\") pod \"maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:34.922518 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:34.922494 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:35.227103 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:35.227081 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65"] Nov 03 14:21:35.228784 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:21:35.228756 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846a80aa_400e_4062_a2b5_057ea0124554.slice/crio-9e76d7bad60c9389ca4bc0ee2849fe627b2f1f54f55b87a030b2a2b73dd0d811 WatchSource:0}: Error finding container 9e76d7bad60c9389ca4bc0ee2849fe627b2f1f54f55b87a030b2a2b73dd0d811: Status 404 returned error can't find the container with id 9e76d7bad60c9389ca4bc0ee2849fe627b2f1f54f55b87a030b2a2b73dd0d811 Nov 03 14:21:36.218900 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:36.218854 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" event={"ID":"846a80aa-400e-4062-a2b5-057ea0124554","Type":"ContainerStarted","Data":"19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11"} Nov 03 14:21:36.219350 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:36.218922 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" event={"ID":"846a80aa-400e-4062-a2b5-057ea0124554","Type":"ContainerStarted","Data":"9e76d7bad60c9389ca4bc0ee2849fe627b2f1f54f55b87a030b2a2b73dd0d811"} Nov 03 14:21:36.219350 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:36.218986 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:36.265247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:36.265184 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" podStartSLOduration=1.921444054 podStartE2EDuration="2.265170891s" totalImagesPullingTime="343.726837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:36.26445385 +0000 UTC m=+571.823090333" watchObservedRunningTime="2026-11-03 14:21:36.265170891 +0000 UTC m=+571.823807372" Nov 03 14:21:36.645449 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:36.645356 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:36.659351 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:36.659323 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:37.235922 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:37.235890 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:21:40.193152 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:40.193120 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" Nov 03 14:21:42.229723 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:42.229695 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:21:50.138741 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.138711 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw"] Nov 03 14:21:50.141419 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.141398 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" Nov 03 14:21:50.165011 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.164985 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw"] Nov 03 14:21:50.172557 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.172530 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-1ffaa0-dockercfg-qhmkv\"" Nov 03 14:21:50.172790 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.172579 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-1ffaa0\"" Nov 03 14:21:50.243721 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.243693 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88"] Nov 03 14:21:50.246855 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.246838 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" Nov 03 14:21:50.268659 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.268634 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88"] Nov 03 14:21:50.300059 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.300014 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnz5\" (UniqueName: \"kubernetes.io/projected/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-kube-api-access-ncnz5\") pod \"payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw\" (UID: \"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" Nov 03 14:21:50.300250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.300083 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw\" (UID: \"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" Nov 03 14:21:50.401162 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.401084 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnz5\" (UniqueName: \"kubernetes.io/projected/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-kube-api-access-ncnz5\") pod \"payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw\" (UID: \"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" Nov 03 14:21:50.401162 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.401128 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5989m\" (UniqueName: \"kubernetes.io/projected/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-kube-api-access-5989m\") pod \"payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88\" (UID: \"2ce7aee1-fade-4b22-bf65-c3e6e6119db8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" Nov 03 14:21:50.401162 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.401156 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw\" (UID: \"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" Nov 03 14:21:50.401392 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.401204 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88\" (UID: \"2ce7aee1-fade-4b22-bf65-c3e6e6119db8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" Nov 03 14:21:50.401922 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.401897 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw\" (UID: \"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" Nov 03 14:21:50.423343 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.423321 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnz5\" (UniqueName: \"kubernetes.io/projected/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-kube-api-access-ncnz5\") pod \"payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw\" (UID: \"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" Nov 03 14:21:50.454586 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.454556 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" Nov 03 14:21:50.502991 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.502950 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88\" (UID: \"2ce7aee1-fade-4b22-bf65-c3e6e6119db8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" Nov 03 14:21:50.503184 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.503111 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5989m\" (UniqueName: \"kubernetes.io/projected/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-kube-api-access-5989m\") pod \"payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88\" (UID: \"2ce7aee1-fade-4b22-bf65-c3e6e6119db8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" Nov 03 14:21:50.506048 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.503877 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88\" (UID: \"2ce7aee1-fade-4b22-bf65-c3e6e6119db8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" Nov 03 14:21:50.527689 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.527637 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5989m\" (UniqueName: \"kubernetes.io/projected/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-kube-api-access-5989m\") pod \"payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88\" (UID: \"2ce7aee1-fade-4b22-bf65-c3e6e6119db8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" Nov 03 14:21:50.562204 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.562163 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" Nov 03 14:21:50.764659 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.764629 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw"] Nov 03 14:21:50.766061 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:21:50.766036 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa8a4f0_30e5_4c0b_9d31_95911c3f0de6.slice/crio-400a3e71c6ee9e8a9a6a34196dc667dc61ff5b1df11fe96512fe41ede293d0e0 WatchSource:0}: Error finding container 400a3e71c6ee9e8a9a6a34196dc667dc61ff5b1df11fe96512fe41ede293d0e0: Status 404 returned error can't find the container with id 400a3e71c6ee9e8a9a6a34196dc667dc61ff5b1df11fe96512fe41ede293d0e0 Nov 03 14:21:50.893351 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:50.893329 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88"] Nov 03 14:21:50.895735 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:21:50.895706 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce7aee1_fade_4b22_bf65_c3e6e6119db8.slice/crio-bf725c1b75875db317d2c5ab0ecc08bf93d14ef580774bc7ec74dbab423d6d7a WatchSource:0}: Error finding container bf725c1b75875db317d2c5ab0ecc08bf93d14ef580774bc7ec74dbab423d6d7a: Status 404 returned error can't find the container with id bf725c1b75875db317d2c5ab0ecc08bf93d14ef580774bc7ec74dbab423d6d7a Nov 03 14:21:51.324839 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:51.324785 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" event={"ID":"2ce7aee1-fade-4b22-bf65-c3e6e6119db8","Type":"ContainerStarted","Data":"152948f06ef3372801f448168e4cc34678ff250bbe2123cfc268fca4ab80e0dc"} Nov 03 14:21:51.325768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:51.325743 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" Nov 03 14:21:51.325899 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:51.325774 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" event={"ID":"2ce7aee1-fade-4b22-bf65-c3e6e6119db8","Type":"ContainerStarted","Data":"bf725c1b75875db317d2c5ab0ecc08bf93d14ef580774bc7ec74dbab423d6d7a"} Nov 03 14:21:51.326634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:51.326592 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" event={"ID":"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6","Type":"ContainerStarted","Data":"823359b8ac65b6ff416da9649eaa08ed39c036abaa263d6d8ad7d8281e9b5b05"} Nov 03 14:21:51.326715 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:51.326648 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" event={"ID":"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6","Type":"ContainerStarted","Data":"400a3e71c6ee9e8a9a6a34196dc667dc61ff5b1df11fe96512fe41ede293d0e0"} Nov 03 14:21:51.326778 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:51.326762 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" Nov 03 14:21:51.372266 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:51.372209 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" podStartSLOduration=1.372196136 podStartE2EDuration="1.372196136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:51.371118248 +0000 UTC m=+586.929754732" watchObservedRunningTime="2026-11-03 14:21:51.372196136 +0000 UTC m=+586.930832620" Nov 03 14:21:51.416721 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:51.416675 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" podStartSLOduration=1.416661806 podStartE2EDuration="1.416661806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:51.414795968 +0000 UTC m=+586.973432453" watchObservedRunningTime="2026-11-03 14:21:51.416661806 +0000 UTC m=+586.975298290" Nov 03 14:21:58.076792 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.076759 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc"] Nov 03 14:21:58.080809 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.080790 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.111149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.111123 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-1ffaa0-serving-cert\"" Nov 03 14:21:58.113912 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.113886 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc"] Nov 03 14:21:58.186661 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.186601 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.186835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.186673 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.186835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.186744 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ce741-6676-438e-933b-47a0a19ee593-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.186835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.186781 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qzxh\" (UniqueName: \"kubernetes.io/projected/2d0ce741-6676-438e-933b-47a0a19ee593-kube-api-access-9qzxh\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.288212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.288171 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.288407 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.288219 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.288407 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.288265 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ce741-6676-438e-933b-47a0a19ee593-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.288407 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.288306 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qzxh\" (UniqueName: \"kubernetes.io/projected/2d0ce741-6676-438e-933b-47a0a19ee593-kube-api-access-9qzxh\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.288407 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:21:58.288352 6389 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-serving-cert: secret "maas-api-e2e-shared-a-w4-1ffaa0-serving-cert" not found Nov 03 14:21:58.288683 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:21:58.288450 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-maas-api-tls podName:2d0ce741-6676-438e-933b-47a0a19ee593 nodeName:}" failed. No retries permitted until 2026-11-03 14:21:58.788425802 +0000 UTC m=+594.347062281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-maas-api-tls") pod "maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" (UID: "2d0ce741-6676-438e-933b-47a0a19ee593") : secret "maas-api-e2e-shared-a-w4-1ffaa0-serving-cert" not found Nov 03 14:21:58.289785 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.289630 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ce741-6676-438e-933b-47a0a19ee593-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.295515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.294167 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.309889 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.309855 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qzxh\" (UniqueName: \"kubernetes.io/projected/2d0ce741-6676-438e-933b-47a0a19ee593-kube-api-access-9qzxh\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.376250 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:21:58.376149 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:21:58.794302 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.794273 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.797003 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.796974 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:58.995095 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:58.995061 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:21:59.302416 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:59.302391 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc"] Nov 03 14:21:59.304265 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:21:59.304235 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0ce741_6676_438e_933b_47a0a19ee593.slice/crio-5ca19d9726d5dbd25c4d9acdab747f185d208b55b2158f1774396ea34160af93 WatchSource:0}: Error finding container 5ca19d9726d5dbd25c4d9acdab747f185d208b55b2158f1774396ea34160af93: Status 404 returned error can't find the container with id 5ca19d9726d5dbd25c4d9acdab747f185d208b55b2158f1774396ea34160af93 Nov 03 14:21:59.372196 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:21:59.372160 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" event={"ID":"2d0ce741-6676-438e-933b-47a0a19ee593","Type":"ContainerStarted","Data":"5ca19d9726d5dbd25c4d9acdab747f185d208b55b2158f1774396ea34160af93"} Nov 03 14:22:00.379379 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:00.379343 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" event={"ID":"2d0ce741-6676-438e-933b-47a0a19ee593","Type":"ContainerStarted","Data":"9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95"} Nov 03 14:22:00.379844 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:00.379548 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:22:00.428666 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:00.428602 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" podStartSLOduration=2.017158221 podStartE2EDuration="2.428588479s" totalImagesPullingTime="411.430258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:00.426773678 +0000 UTC m=+595.985410162" watchObservedRunningTime="2026-11-03 14:22:00.428588479 +0000 UTC m=+595.987224964" Nov 03 14:22:02.338372 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:02.338331 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" Nov 03 14:22:03.343411 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:03.343380 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" Nov 03 14:22:06.396666 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:06.392907 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:22:16.258371 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.258341 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6"] Nov 03 14:22:16.262253 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.262232 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.282335 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.282301 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6"] Nov 03 14:22:16.292437 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.292403 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"openshift-service-ca.crt\"" Nov 03 14:22:16.292437 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.292418 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"default-dockercfg-wqn4r\"" Nov 03 14:22:16.292691 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.292451 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"kube-root-ca.crt\"" Nov 03 14:22:16.292691 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.292503 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"e2e-conflict-a-1c60a22c-kserve-self-signed-certs\"" Nov 03 14:22:16.388625 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.388585 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-model-cache\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.388793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.388706 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvp8\" (UniqueName: \"kubernetes.io/projected/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-kube-api-access-jdvp8\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.388793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.388746 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tmp-dir\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.388793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.388776 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-home\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.388898 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.388822 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tls-certs\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.388932 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.388896 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-dshm\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.491702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.491652 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-model-cache\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.491891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.491743 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvp8\" (UniqueName: \"kubernetes.io/projected/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-kube-api-access-jdvp8\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.491891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.491785 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tmp-dir\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.491891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.491828 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-home\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.491891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.491877 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tls-certs\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.492122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.491907 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-dshm\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.492700 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.492669 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-model-cache\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.492972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.492943 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-home\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.493579 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.493556 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tmp-dir\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.496642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.496598 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-dshm\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.499423 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.499396 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tls-certs\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.520733 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.520674 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvp8\" (UniqueName: \"kubernetes.io/projected/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-kube-api-access-jdvp8\") pod \"e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.577184 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.577159 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:16.882139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:16.882109 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6"] Nov 03 14:22:16.883823 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:22:16.883796 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc34cebc8_7ace_4844_a234_4f2d0a8a3c76.slice/crio-df746478332140e766b2857a67eadb87ada86df9323fdb585f1e4570df56b36f WatchSource:0}: Error finding container df746478332140e766b2857a67eadb87ada86df9323fdb585f1e4570df56b36f: Status 404 returned error can't find the container with id df746478332140e766b2857a67eadb87ada86df9323fdb585f1e4570df56b36f Nov 03 14:22:17.494721 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:17.494682 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" event={"ID":"c34cebc8-7ace-4844-a234-4f2d0a8a3c76","Type":"ContainerStarted","Data":"87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40"} Nov 03 14:22:17.495122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:17.494730 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" event={"ID":"c34cebc8-7ace-4844-a234-4f2d0a8a3c76","Type":"ContainerStarted","Data":"df746478332140e766b2857a67eadb87ada86df9323fdb585f1e4570df56b36f"} Nov 03 14:22:17.543493 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:17.543440 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" podStartSLOduration=1.543427226 podStartE2EDuration="1.543427226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:17.541686602 +0000 UTC m=+613.100323087" watchObservedRunningTime="2026-11-03 14:22:17.543427226 +0000 UTC m=+613.102063709" Nov 03 14:22:17.578163 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:17.578132 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:26.578333 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:26.578298 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:26.591499 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:26.591472 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:27.556994 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:27.556963 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:22:28.374203 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:22:28.374168 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:22:30.568956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.568927 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26"] Nov 03 14:22:30.573822 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.573805 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:30.594960 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.594930 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26"] Nov 03 14:22:30.604641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.604423 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-2d0da2-serving-cert\"" Nov 03 14:22:30.667689 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.667656 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c918c5c-994d-4531-ad18-89ee658e2c4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:30.667689 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.667693 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:30.667943 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.667854 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:30.667943 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.667897 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt4cl\" (UniqueName: \"kubernetes.io/projected/0c918c5c-994d-4531-ad18-89ee658e2c4e-kube-api-access-tt4cl\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:30.774286 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.774240 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c918c5c-994d-4531-ad18-89ee658e2c4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:30.774482 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.774299 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:30.774482 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.774411 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:30.776820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.776788 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt4cl\" (UniqueName: \"kubernetes.io/projected/0c918c5c-994d-4531-ad18-89ee658e2c4e-kube-api-access-tt4cl\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:30.776994 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:22:30.776336 6389 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-serving-cert: secret "maas-api-e2e-shared-b-w4-2d0da2-serving-cert" not found Nov 03 14:22:30.777065 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.777040 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c918c5c-994d-4531-ad18-89ee658e2c4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:30.777128 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:22:30.777096 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-maas-api-tls podName:0c918c5c-994d-4531-ad18-89ee658e2c4e nodeName:}" failed. No retries permitted until 2026-11-03 14:22:31.277067804 +0000 UTC m=+626.835704278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-maas-api-tls") pod "maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" (UID: "0c918c5c-994d-4531-ad18-89ee658e2c4e") : secret "maas-api-e2e-shared-b-w4-2d0da2-serving-cert" not found Nov 03 14:22:30.786040 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.781373 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:30.801926 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:30.801900 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt4cl\" (UniqueName: \"kubernetes.io/projected/0c918c5c-994d-4531-ad18-89ee658e2c4e-kube-api-access-tt4cl\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:31.281256 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:31.281218 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:31.283809 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:31.283786 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:31.487461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:31.487426 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:31.791794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:31.791771 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26"] Nov 03 14:22:31.793748 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:22:31.793717 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c918c5c_994d_4531_ad18_89ee658e2c4e.slice/crio-cf01a144bf0a1dbe7cbf48fd5ab3a1e620ae25c6ab0028c4c3d4f1586e5f1674 WatchSource:0}: Error finding container cf01a144bf0a1dbe7cbf48fd5ab3a1e620ae25c6ab0028c4c3d4f1586e5f1674: Status 404 returned error can't find the container with id cf01a144bf0a1dbe7cbf48fd5ab3a1e620ae25c6ab0028c4c3d4f1586e5f1674 Nov 03 14:22:32.598398 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:32.598307 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" event={"ID":"0c918c5c-994d-4531-ad18-89ee658e2c4e","Type":"ContainerStarted","Data":"82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a"} Nov 03 14:22:32.598398 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:32.598383 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" event={"ID":"0c918c5c-994d-4531-ad18-89ee658e2c4e","Type":"ContainerStarted","Data":"cf01a144bf0a1dbe7cbf48fd5ab3a1e620ae25c6ab0028c4c3d4f1586e5f1674"} Nov 03 14:22:32.598590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:32.598426 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:32.643070 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:32.643021 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" podStartSLOduration=2.237340978 podStartE2EDuration="2.643007299s" totalImagesPullingTime="405.666321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:32.641269126 +0000 UTC m=+628.199905611" watchObservedRunningTime="2026-11-03 14:22:32.643007299 +0000 UTC m=+628.201643804" Nov 03 14:22:38.617197 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:38.617167 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:22:40.124714 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.124674 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc"] Nov 03 14:22:40.129123 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.129098 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.145963 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.145942 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-156ae1-openshift-default-dockercfg-hh5m9\"" Nov 03 14:22:40.164550 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.164516 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc"] Nov 03 14:22:40.180229 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.180185 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.180412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.180270 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.180412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.180311 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.180412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.180388 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.180562 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.180448 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.180562 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.180506 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.180693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.180562 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.180693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.180627 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d9km\" (UniqueName: \"kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-kube-api-access-8d9km\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.180693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.180677 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.281835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.281797 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282013 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.281850 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282013 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.281882 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282013 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.281913 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282013 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.281939 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d9km\" (UniqueName: \"kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-kube-api-access-8d9km\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282013 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.281975 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.282015 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.282061 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.282089 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.282255 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.282289 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.282314 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282701 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.282554 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.282944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.282833 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.284591 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.284569 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.284713 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.284695 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.303696 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.303662 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.303833 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.303700 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d9km\" (UniqueName: \"kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-kube-api-access-8d9km\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.442587 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.442551 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:40.761089 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.761063 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc"] Nov 03 14:22:40.762984 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:22:40.762957 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1b2c5f_ad7b_4a8b_a6b7_6b0082106ca5.slice/crio-54a55b3a7b5e02c2d5233d1ffb92f4004e7ace6dd2b7bb89124f983558225f96 WatchSource:0}: Error finding container 54a55b3a7b5e02c2d5233d1ffb92f4004e7ace6dd2b7bb89124f983558225f96: Status 404 returned error can't find the container with id 54a55b3a7b5e02c2d5233d1ffb92f4004e7ace6dd2b7bb89124f983558225f96 Nov 03 14:22:40.765505 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.765475 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:22:40.765597 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.765540 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:22:40.765597 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:40.765570 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:22:41.653755 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:41.653723 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" event={"ID":"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5","Type":"ContainerStarted","Data":"917af16a05d736cb6a02b1140ceda5c4aff93ba42769990c370303e4a9444a85"} Nov 03 14:22:41.654261 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:41.653773 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" event={"ID":"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5","Type":"ContainerStarted","Data":"54a55b3a7b5e02c2d5233d1ffb92f4004e7ace6dd2b7bb89124f983558225f96"} Nov 03 14:22:41.714414 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:41.714361 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" podStartSLOduration=1.714343461 podStartE2EDuration="1.714343461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:41.712951591 +0000 UTC m=+637.271588076" watchObservedRunningTime="2026-11-03 14:22:41.714343461 +0000 UTC m=+637.272979946" Nov 03 14:22:42.443854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:42.443816 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:42.445417 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:42.445385 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:42.445583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:42.445450 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" podUID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 14:22:43.444123 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:43.444027 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:43.444123 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:43.444084 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" podUID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 14:22:44.443632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:44.443015 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:44.443632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:44.443079 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" podUID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 14:22:45.443642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:45.443004 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:45.443642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:45.443066 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" podUID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 14:22:46.443071 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:46.443028 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:46.443442 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:46.443409 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" podUID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 14:22:47.442857 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:47.442818 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:47.443423 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:47.442889 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" podUID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 14:22:49.484801 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:49.484763 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:22:49.485306 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:49.484823 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" podUID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:22:50.043288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:50.043257 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:50.043748 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:50.043722 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:50.044571 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:22:50.044554 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:22:58.374282 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:22:58.374246 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:23:04.160121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:04.160088 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6"] Nov 03 14:23:19.181888 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.181844 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" podUID="c34cebc8-7ace-4844-a234-4f2d0a8a3c76" containerName="main" containerID="cri-o://87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40" gracePeriod=45 Nov 03 14:23:19.452789 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.452763 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:23:19.597834 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.597802 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-kube-api-access-jdvp8\" (UniqueName: \"kubernetes.io/projected/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-kube-api-access-jdvp8\") pod \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " Nov 03 14:23:19.598007 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.597870 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tls-certs\" (UniqueName: \"kubernetes.io/secret/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tls-certs\") pod \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " Nov 03 14:23:19.598007 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.597905 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-model-cache\") pod \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " Nov 03 14:23:19.598080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.598052 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tmp-dir\") pod \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " Nov 03 14:23:19.598138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.598122 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-home\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-home\") pod \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " Nov 03 14:23:19.598186 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.598162 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-dshm\") pod \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\" (UID: \"c34cebc8-7ace-4844-a234-4f2d0a8a3c76\") " Nov 03 14:23:19.598240 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.598188 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-model-cache" pod "c34cebc8-7ace-4844-a234-4f2d0a8a3c76" (UID: "c34cebc8-7ace-4844-a234-4f2d0a8a3c76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:19.598334 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.598301 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tmp-dir" pod "c34cebc8-7ace-4844-a234-4f2d0a8a3c76" (UID: "c34cebc8-7ace-4844-a234-4f2d0a8a3c76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:19.598425 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.598402 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-home" pod "c34cebc8-7ace-4844-a234-4f2d0a8a3c76" (UID: "c34cebc8-7ace-4844-a234-4f2d0a8a3c76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:19.598652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.598636 6389 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-model-cache\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:23:19.598732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.598660 6389 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tmp-dir\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:23:19.598732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.598673 6389 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-home\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:23:19.600262 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.600233 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-dshm" pod "c34cebc8-7ace-4844-a234-4f2d0a8a3c76" (UID: "c34cebc8-7ace-4844-a234-4f2d0a8a3c76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:19.600376 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.600277 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tls-certs" pod "c34cebc8-7ace-4844-a234-4f2d0a8a3c76" (UID: "c34cebc8-7ace-4844-a234-4f2d0a8a3c76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:23:19.600376 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.600344 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-kube-api-access-jdvp8" pod "c34cebc8-7ace-4844-a234-4f2d0a8a3c76" (UID: "c34cebc8-7ace-4844-a234-4f2d0a8a3c76"). InnerVolumeSpecName "kube-api-access-jdvp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:19.699967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.699885 6389 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-dshm\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:23:19.699967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.699913 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdvp8\" (UniqueName: \"kubernetes.io/projected/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-kube-api-access-jdvp8\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:23:19.699967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.699926 6389 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34cebc8-7ace-4844-a234-4f2d0a8a3c76-tls-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:23:19.913519 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.913485 6389 generic.go:356] "Generic (PLEG): container finished" podID="c34cebc8-7ace-4844-a234-4f2d0a8a3c76" containerID="87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40" exitCode=0 Nov 03 14:23:19.913710 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.913534 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" Nov 03 14:23:19.913710 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.913572 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" event={"ID":"c34cebc8-7ace-4844-a234-4f2d0a8a3c76","Type":"ContainerDied","Data":"87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40"} Nov 03 14:23:19.913710 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.913638 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6" event={"ID":"c34cebc8-7ace-4844-a234-4f2d0a8a3c76","Type":"ContainerDied","Data":"df746478332140e766b2857a67eadb87ada86df9323fdb585f1e4570df56b36f"} Nov 03 14:23:19.913710 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.913661 6389 scope.go:122] "RemoveContainer" containerID="87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40" Nov 03 14:23:19.925572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.925549 6389 scope.go:122] "RemoveContainer" containerID="87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40" Nov 03 14:23:19.925895 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:23:19.925873 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40\": container with ID starting with 87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40 not found: ID does not exist" containerID="87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40" Nov 03 14:23:19.925958 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.925903 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40"} err="failed to get container status \"87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40\": rpc error: code = NotFound desc = could not find container \"87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40\": container with ID starting with 87a8d7da7f1c44f1b3b74ae839fa3ba06f9fef2c1e3064f09a1dafe58df1cd40 not found: ID does not exist" Nov 03 14:23:19.964571 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.964497 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6"] Nov 03 14:23:19.966461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:19.966438 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/e2e-conflict-a-1c60a22c-kserve-5b9f9545b9-2cvg6"] Nov 03 14:23:20.621252 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.621188 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34cebc8-7ace-4844-a234-4f2d0a8a3c76" path="/var/lib/kubelet/pods/c34cebc8-7ace-4844-a234-4f2d0a8a3c76/volumes" Nov 03 14:23:20.775751 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.775716 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59"] Nov 03 14:23:20.776253 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.776241 6389 cpu_manager.go:414] "containerMap: removing container" podUID="c34cebc8-7ace-4844-a234-4f2d0a8a3c76" containerName="main" Nov 03 14:23:20.776296 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.776256 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34cebc8-7ace-4844-a234-4f2d0a8a3c76" containerName="main" Nov 03 14:23:20.776391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.776359 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34cebc8-7ace-4844-a234-4f2d0a8a3c76" containerName="main" Nov 03 14:23:20.780990 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.780971 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" Nov 03 14:23:20.799008 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.798982 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59"] Nov 03 14:23:20.807559 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.807534 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5b97f1\"" Nov 03 14:23:20.807738 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.807572 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5b97f1-dockercfg-lc2t7\"" Nov 03 14:23:20.870023 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.869987 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t"] Nov 03 14:23:20.892772 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.891899 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" Nov 03 14:23:20.893059 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.893033 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t"] Nov 03 14:23:20.913703 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.913672 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcb7p\" (UniqueName: \"kubernetes.io/projected/2142f874-bf66-4f25-8b7b-f26204387a6c-kube-api-access-tcb7p\") pod \"payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59\" (UID: \"2142f874-bf66-4f25-8b7b-f26204387a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" Nov 03 14:23:20.913991 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:20.913973 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2142f874-bf66-4f25-8b7b-f26204387a6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59\" (UID: \"2142f874-bf66-4f25-8b7b-f26204387a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" Nov 03 14:23:21.017309 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.017274 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2142f874-bf66-4f25-8b7b-f26204387a6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59\" (UID: \"2142f874-bf66-4f25-8b7b-f26204387a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" Nov 03 14:23:21.017495 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.017336 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbzq4\" (UniqueName: \"kubernetes.io/projected/2d7ec965-e58c-423a-9772-5d3cfa5063a5-kube-api-access-tbzq4\") pod \"payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t\" (UID: \"2d7ec965-e58c-423a-9772-5d3cfa5063a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" Nov 03 14:23:21.017495 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.017354 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7ec965-e58c-423a-9772-5d3cfa5063a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t\" (UID: \"2d7ec965-e58c-423a-9772-5d3cfa5063a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" Nov 03 14:23:21.017495 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.017414 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcb7p\" (UniqueName: \"kubernetes.io/projected/2142f874-bf66-4f25-8b7b-f26204387a6c-kube-api-access-tcb7p\") pod \"payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59\" (UID: \"2142f874-bf66-4f25-8b7b-f26204387a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" Nov 03 14:23:21.018163 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.018136 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2142f874-bf66-4f25-8b7b-f26204387a6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59\" (UID: \"2142f874-bf66-4f25-8b7b-f26204387a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" Nov 03 14:23:21.041363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.041335 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcb7p\" (UniqueName: \"kubernetes.io/projected/2142f874-bf66-4f25-8b7b-f26204387a6c-kube-api-access-tcb7p\") pod \"payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59\" (UID: \"2142f874-bf66-4f25-8b7b-f26204387a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" Nov 03 14:23:21.094719 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.094686 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" Nov 03 14:23:21.119160 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.119125 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbzq4\" (UniqueName: \"kubernetes.io/projected/2d7ec965-e58c-423a-9772-5d3cfa5063a5-kube-api-access-tbzq4\") pod \"payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t\" (UID: \"2d7ec965-e58c-423a-9772-5d3cfa5063a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" Nov 03 14:23:21.119330 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.119175 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7ec965-e58c-423a-9772-5d3cfa5063a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t\" (UID: \"2d7ec965-e58c-423a-9772-5d3cfa5063a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" Nov 03 14:23:21.120037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.120013 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7ec965-e58c-423a-9772-5d3cfa5063a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t\" (UID: \"2d7ec965-e58c-423a-9772-5d3cfa5063a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" Nov 03 14:23:21.142264 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.142222 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbzq4\" (UniqueName: \"kubernetes.io/projected/2d7ec965-e58c-423a-9772-5d3cfa5063a5-kube-api-access-tbzq4\") pod \"payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t\" (UID: \"2d7ec965-e58c-423a-9772-5d3cfa5063a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" Nov 03 14:23:21.222686 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.222655 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" Nov 03 14:23:21.402292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.402242 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59"] Nov 03 14:23:21.404003 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:23:21.403973 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2142f874_bf66_4f25_8b7b_f26204387a6c.slice/crio-d71a3fd9802f527b74e69744b8ddfc026ad7db5510c4bff691de61e23c8e71b8 WatchSource:0}: Error finding container d71a3fd9802f527b74e69744b8ddfc026ad7db5510c4bff691de61e23c8e71b8: Status 404 returned error can't find the container with id d71a3fd9802f527b74e69744b8ddfc026ad7db5510c4bff691de61e23c8e71b8 Nov 03 14:23:21.528637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.528588 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t"] Nov 03 14:23:21.529703 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:23:21.529678 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7ec965_e58c_423a_9772_5d3cfa5063a5.slice/crio-78a6e831738ccb53af799c64ed93a0a5e1408136717cde17f3f50c1ddfe54c4f WatchSource:0}: Error finding container 78a6e831738ccb53af799c64ed93a0a5e1408136717cde17f3f50c1ddfe54c4f: Status 404 returned error can't find the container with id 78a6e831738ccb53af799c64ed93a0a5e1408136717cde17f3f50c1ddfe54c4f Nov 03 14:23:21.933443 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.933399 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" event={"ID":"2142f874-bf66-4f25-8b7b-f26204387a6c","Type":"ContainerStarted","Data":"54f5934287357f058dcfe10839fcb1a97a3045c02e1e4b6fff9d8b0aedf73d87"} Nov 03 14:23:21.933947 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.933472 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" event={"ID":"2142f874-bf66-4f25-8b7b-f26204387a6c","Type":"ContainerStarted","Data":"d71a3fd9802f527b74e69744b8ddfc026ad7db5510c4bff691de61e23c8e71b8"} Nov 03 14:23:21.933947 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.933509 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" Nov 03 14:23:21.935115 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.935083 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" event={"ID":"2d7ec965-e58c-423a-9772-5d3cfa5063a5","Type":"ContainerStarted","Data":"0278a161be0a79daf5f8f1978a4d504532979028d55b0b1cf6be1f1febbe8980"} Nov 03 14:23:21.935230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.935127 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" event={"ID":"2d7ec965-e58c-423a-9772-5d3cfa5063a5","Type":"ContainerStarted","Data":"78a6e831738ccb53af799c64ed93a0a5e1408136717cde17f3f50c1ddfe54c4f"} Nov 03 14:23:21.935230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.935148 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" Nov 03 14:23:21.974432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:21.974375 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" podStartSLOduration=1.9743611479999998 podStartE2EDuration="1.974361148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:21.972778563 +0000 UTC m=+677.531415048" watchObservedRunningTime="2026-11-03 14:23:21.974361148 +0000 UTC m=+677.532997673" Nov 03 14:23:22.013487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:22.013435 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" podStartSLOduration=2.013422282 podStartE2EDuration="2.013422282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:22.011863994 +0000 UTC m=+677.570500502" watchObservedRunningTime="2026-11-03 14:23:22.013422282 +0000 UTC m=+677.572058766" Nov 03 14:23:25.085202 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.085105 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd"] Nov 03 14:23:25.089117 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.089087 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.105398 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.105378 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-86c2e2-openshift-default-dockercfg-7dv7n\"" Nov 03 14:23:25.126308 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.125289 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd"] Nov 03 14:23:25.265564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.265517 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.265564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.265562 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.265800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.265596 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzkr\" (UniqueName: \"kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-kube-api-access-7zzkr\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.265800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.265767 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30003229-866c-4a72-be45-37c3134e3159-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.265922 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.265826 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30003229-866c-4a72-be45-37c3134e3159-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.265922 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.265905 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.266066 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.266015 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.266066 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.266048 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.266184 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.266065 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.367707 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.367597 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.367707 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.367673 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.367707 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.367706 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzkr\" (UniqueName: \"kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-kube-api-access-7zzkr\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.367971 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.367743 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30003229-866c-4a72-be45-37c3134e3159-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.367971 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.367778 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30003229-866c-4a72-be45-37c3134e3159-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.367971 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.367809 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.367971 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.367874 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.367971 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.367892 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.367971 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.367911 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.368287 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.368050 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.368287 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.368105 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.368407 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.368306 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.368407 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.368336 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.368545 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.368521 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30003229-866c-4a72-be45-37c3134e3159-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.370321 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.370301 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.370431 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.370341 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30003229-866c-4a72-be45-37c3134e3159-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.387996 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.387968 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.388287 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.388269 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzkr\" (UniqueName: \"kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-kube-api-access-7zzkr\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.404149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.404121 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:25.723172 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.723124 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd"] Nov 03 14:23:25.725998 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:23:25.725974 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30003229_866c_4a72_be45_37c3134e3159.slice/crio-128f716c5479d4cbf31abd3c366c9c0b991b87f75fd28408278c9403e809fbf1 WatchSource:0}: Error finding container 128f716c5479d4cbf31abd3c366c9c0b991b87f75fd28408278c9403e809fbf1: Status 404 returned error can't find the container with id 128f716c5479d4cbf31abd3c366c9c0b991b87f75fd28408278c9403e809fbf1 Nov 03 14:23:25.729183 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.729152 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:23:25.729276 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.729220 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:23:25.729276 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.729247 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:23:25.957703 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.957670 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" event={"ID":"30003229-866c-4a72-be45-37c3134e3159","Type":"ContainerStarted","Data":"78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07"} Nov 03 14:23:25.957856 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:25.957726 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" event={"ID":"30003229-866c-4a72-be45-37c3134e3159","Type":"ContainerStarted","Data":"128f716c5479d4cbf31abd3c366c9c0b991b87f75fd28408278c9403e809fbf1"} Nov 03 14:23:26.028800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:26.028664 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" podStartSLOduration=1.028641977 podStartE2EDuration="1.028641977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:26.003758577 +0000 UTC m=+681.562395083" watchObservedRunningTime="2026-11-03 14:23:26.028641977 +0000 UTC m=+681.587278466" Nov 03 14:23:26.404657 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:26.404562 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:26.406471 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:26.406448 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 14:23:26.406582 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:26.406499 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" podUID="30003229-866c-4a72-be45-37c3134e3159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 14:23:27.405287 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:27.405258 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 14:23:27.405713 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:27.405304 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" podUID="30003229-866c-4a72-be45-37c3134e3159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 14:23:28.375445 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:23:28.375408 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:23:28.405203 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:28.405172 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 14:23:28.405335 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:28.405226 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" podUID="30003229-866c-4a72-be45-37c3134e3159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 14:23:29.404905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:29.404874 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 14:23:29.405193 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:29.404932 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" podUID="30003229-866c-4a72-be45-37c3134e3159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 14:23:30.404854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:30.404817 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 14:23:30.405244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:30.404884 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" podUID="30003229-866c-4a72-be45-37c3134e3159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 14:23:31.405373 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:31.405341 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 14:23:31.405783 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:31.405393 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" podUID="30003229-866c-4a72-be45-37c3134e3159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 14:23:32.405230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:32.405196 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 14:23:32.405420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:32.405247 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" podUID="30003229-866c-4a72-be45-37c3134e3159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 14:23:32.943315 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:32.943290 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" Nov 03 14:23:32.943486 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:32.943351 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" Nov 03 14:23:33.667632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:33.666668 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:33.667632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:33.667059 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:33.668158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:33.667984 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:23:33.864479 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:33.864444 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx"] Nov 03 14:23:33.868496 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:33.868477 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:33.888989 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:33.888961 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5b97f1-serving-cert\"" Nov 03 14:23:33.891237 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:33.891213 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx"] Nov 03 14:23:33.912775 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:33.912742 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:33.913092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:33.912886 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-maas-api-tls\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:33.913092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:33.912924 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gzz\" (UniqueName: \"kubernetes.io/projected/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-kube-api-access-x2gzz\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:33.913092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:33.913008 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-metrics-certs\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:34.014672 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:34.014582 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-maas-api-tls\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:34.014672 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:34.014640 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gzz\" (UniqueName: \"kubernetes.io/projected/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-kube-api-access-x2gzz\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:34.014904 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:34.014706 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-metrics-certs\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:34.014904 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:23:34.014736 6389 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-serving-cert: secret "maas-api-e2e-inf-a-5b97f1-serving-cert" not found Nov 03 14:23:34.014904 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:23:34.014798 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-maas-api-tls podName:dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b nodeName:}" failed. No retries permitted until 2026-11-03 14:23:34.5147838 +0000 UTC m=+690.073420261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-maas-api-tls") pod "maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" (UID: "dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b") : secret "maas-api-e2e-inf-a-5b97f1-serving-cert" not found Nov 03 14:23:34.014904 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:34.014831 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:34.015423 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:34.015405 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:34.017163 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:34.017143 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-metrics-certs\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:34.047427 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:34.047392 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gzz\" (UniqueName: \"kubernetes.io/projected/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-kube-api-access-x2gzz\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:34.520109 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:34.520071 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-maas-api-tls\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:34.522641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:34.522600 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-maas-api-tls\") pod \"maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:34.788092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:34.788012 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:35.107713 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:35.107551 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx"] Nov 03 14:23:35.125404 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:23:35.125364 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6f4ffd_b75d_4b26_a2a9_9e9ef051654b.slice/crio-c1bd5389b055b7de9a2d976eda7be7abdc279126ebbb039f42eba9ce5b0dfdb2 WatchSource:0}: Error finding container c1bd5389b055b7de9a2d976eda7be7abdc279126ebbb039f42eba9ce5b0dfdb2: Status 404 returned error can't find the container with id c1bd5389b055b7de9a2d976eda7be7abdc279126ebbb039f42eba9ce5b0dfdb2 Nov 03 14:23:36.045882 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:36.045844 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" event={"ID":"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b","Type":"ContainerStarted","Data":"c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93"} Nov 03 14:23:36.046306 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:36.045908 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" event={"ID":"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b","Type":"ContainerStarted","Data":"c1bd5389b055b7de9a2d976eda7be7abdc279126ebbb039f42eba9ce5b0dfdb2"} Nov 03 14:23:36.046306 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:36.046023 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:36.091539 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:36.091489 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" podStartSLOduration=2.683815986 podStartE2EDuration="3.091475437s" totalImagesPullingTime="407.659451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:36.089926442 +0000 UTC m=+691.648562924" watchObservedRunningTime="2026-11-03 14:23:36.091475437 +0000 UTC m=+691.650111921" Nov 03 14:23:40.319727 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.319678 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb"] Nov 03 14:23:40.323817 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.323795 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.356161 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.356114 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb"] Nov 03 14:23:40.358131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.358107 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4"] Nov 03 14:23:40.362793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.362769 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.383001 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.382969 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4"] Nov 03 14:23:40.416940 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.416900 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.417097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.416951 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.417097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.416991 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.417097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417021 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqlrw\" (UniqueName: \"kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-kube-api-access-lqlrw\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.417097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417067 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.417097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417094 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.417278 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417154 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.417278 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417175 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7m8n\" (UniqueName: \"kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-kube-api-access-k7m8n\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.417278 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417214 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.417278 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417239 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.417278 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417267 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.417542 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417293 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.417542 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417331 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.417673 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417572 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.417673 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417627 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.417673 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417655 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.417823 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417701 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.417823 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.417724 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.518576 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.518544 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.518576 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.518579 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.518849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.518594 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.518849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.518663 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.518849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.518682 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.518849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.518721 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.518849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.518794 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.518849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.518845 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.519154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.518868 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqlrw\" (UniqueName: \"kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-kube-api-access-lqlrw\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.519154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.518916 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.519154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.518947 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.519154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519013 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.519154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519036 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7m8n\" (UniqueName: \"kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-kube-api-access-k7m8n\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.519154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519079 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.519154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519107 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.519154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519138 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.519154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519146 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.519641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519166 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.519641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519174 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.519641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519228 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.519641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519381 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.519641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519438 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.519641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519498 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.519955 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519747 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.519955 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519877 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.520045 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.519965 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.520081 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.520036 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.520671 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.520562 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.521388 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.521366 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.521672 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.521651 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.521849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.521832 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.522063 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.522047 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.539383 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.539343 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.540181 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.540135 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7m8n\" (UniqueName: \"kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-kube-api-access-k7m8n\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.550036 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.542111 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.563287 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.563249 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqlrw\" (UniqueName: \"kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-kube-api-access-lqlrw\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.638261 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.638187 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:40.680323 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.680291 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:40.973826 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.973792 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb"] Nov 03 14:23:40.975225 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:23:40.975188 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a69b4c1_74d0_4515_b0a0_1f7614b0a138.slice/crio-768379f1f4aa4ba1f663490214298bd2e924b42ce202dec3861e33e6b07842e2 WatchSource:0}: Error finding container 768379f1f4aa4ba1f663490214298bd2e924b42ce202dec3861e33e6b07842e2: Status 404 returned error can't find the container with id 768379f1f4aa4ba1f663490214298bd2e924b42ce202dec3861e33e6b07842e2 Nov 03 14:23:40.977566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.977535 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:23:40.977693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.977642 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:23:40.977693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:40.977687 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:23:41.014217 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:41.014195 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4"] Nov 03 14:23:41.015832 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:23:41.015809 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5004ddef_5dbd_4ea1_bb10_bc82a37791b8.slice/crio-691f7779ce6a23e715d332d1524355f9770c3ab7be4e9381aa8017c2be06f27a WatchSource:0}: Error finding container 691f7779ce6a23e715d332d1524355f9770c3ab7be4e9381aa8017c2be06f27a: Status 404 returned error can't find the container with id 691f7779ce6a23e715d332d1524355f9770c3ab7be4e9381aa8017c2be06f27a Nov 03 14:23:41.018194 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:41.018168 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:23:41.018270 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:41.018226 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:23:41.018270 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:41.018256 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:23:41.076565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:41.076504 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" event={"ID":"4a69b4c1-74d0-4515-b0a0-1f7614b0a138","Type":"ContainerStarted","Data":"6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8"} Nov 03 14:23:41.076861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:41.076839 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" event={"ID":"4a69b4c1-74d0-4515-b0a0-1f7614b0a138","Type":"ContainerStarted","Data":"768379f1f4aa4ba1f663490214298bd2e924b42ce202dec3861e33e6b07842e2"} Nov 03 14:23:41.078861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:41.078842 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" event={"ID":"5004ddef-5dbd-4ea1-bb10-bc82a37791b8","Type":"ContainerStarted","Data":"691f7779ce6a23e715d332d1524355f9770c3ab7be4e9381aa8017c2be06f27a"} Nov 03 14:23:41.122135 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:41.122063 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" podStartSLOduration=1.122044837 podStartE2EDuration="1.122044837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:41.120042302 +0000 UTC m=+696.678678785" watchObservedRunningTime="2026-11-03 14:23:41.122044837 +0000 UTC m=+696.680681325" Nov 03 14:23:41.640743 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:41.640694 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:42.070632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:42.070237 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:23:42.087900 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:42.087699 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" event={"ID":"5004ddef-5dbd-4ea1-bb10-bc82a37791b8","Type":"ContainerStarted","Data":"0b84f7b3ca712899e9d4406b06f1f91da610a51f1bc5c91327035fc37cbeb245"} Nov 03 14:23:42.203369 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:42.203304 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" podStartSLOduration=2.203285623 podStartE2EDuration="2.203285623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:42.184055589 +0000 UTC m=+697.742692098" watchObservedRunningTime="2026-11-03 14:23:42.203285623 +0000 UTC m=+697.761922108" Nov 03 14:23:42.641633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:42.641224 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 14:23:42.641633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:42.641287 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" podUID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 14:23:42.684633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:42.684380 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:42.684633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:42.684495 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 14:23:42.684633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:42.684541 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" podUID="5004ddef-5dbd-4ea1-bb10-bc82a37791b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 14:23:43.659929 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:43.648900 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 14:23:43.659929 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:43.648974 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" podUID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 14:23:43.682515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:43.682478 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 14:23:43.682754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:43.682535 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" podUID="5004ddef-5dbd-4ea1-bb10-bc82a37791b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 14:23:44.667636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:44.666095 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 14:23:44.667636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:44.666153 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" podUID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 14:23:44.681069 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:44.681032 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 14:23:44.681249 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:44.681094 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" podUID="5004ddef-5dbd-4ea1-bb10-bc82a37791b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 14:23:45.641630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:45.639955 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 14:23:45.641630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:45.640017 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" podUID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 14:23:45.680759 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:45.680726 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 14:23:45.681145 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:45.680773 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" podUID="5004ddef-5dbd-4ea1-bb10-bc82a37791b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 14:23:46.645635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:46.645150 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 14:23:46.645635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:46.645210 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" podUID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 14:23:46.680985 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:46.680956 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 14:23:46.681425 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:46.681399 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" podUID="5004ddef-5dbd-4ea1-bb10-bc82a37791b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 14:23:47.639413 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:47.639378 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 14:23:47.639587 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:47.639444 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" podUID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 14:23:47.681189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:47.681152 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 14:23:47.681566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:47.681208 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" podUID="5004ddef-5dbd-4ea1-bb10-bc82a37791b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 14:23:48.643926 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:48.643896 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:48.644325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:48.644299 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:48.645057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:48.645039 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:23:48.733022 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:48.732696 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:48.733529 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:48.733508 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:48.734784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:23:48.734753 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:23:58.373993 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:23:58.373954 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:24:00.773497 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:00.773468 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl"] Nov 03 14:24:00.774013 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:00.773706 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" podUID="0952ddc3-f129-4d1e-8d43-c73e01cf8258" containerName="maas-api" containerID="cri-o://8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3" gracePeriod=30 Nov 03 14:24:01.030984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.030922 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:24:01.151632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.146566 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-maas-api-tls\") pod \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " Nov 03 14:24:01.151632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.146640 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0952ddc3-f129-4d1e-8d43-c73e01cf8258-kube-api-access-4vf7c\" (UniqueName: \"kubernetes.io/projected/0952ddc3-f129-4d1e-8d43-c73e01cf8258-kube-api-access-4vf7c\") pod \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " Nov 03 14:24:01.151632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.146692 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-metrics-certs\") pod \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " Nov 03 14:24:01.151632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.146853 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0952ddc3-f129-4d1e-8d43-c73e01cf8258-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0952ddc3-f129-4d1e-8d43-c73e01cf8258-openshift-service-ca\") pod \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\" (UID: \"0952ddc3-f129-4d1e-8d43-c73e01cf8258\") " Nov 03 14:24:01.151632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.147622 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0952ddc3-f129-4d1e-8d43-c73e01cf8258-openshift-service-ca" pod "0952ddc3-f129-4d1e-8d43-c73e01cf8258" (UID: "0952ddc3-f129-4d1e-8d43-c73e01cf8258"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:01.169862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.167386 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0952ddc3-f129-4d1e-8d43-c73e01cf8258-kube-api-access-4vf7c" pod "0952ddc3-f129-4d1e-8d43-c73e01cf8258" (UID: "0952ddc3-f129-4d1e-8d43-c73e01cf8258"). InnerVolumeSpecName "kube-api-access-4vf7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:01.169862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.168401 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-metrics-certs" pod "0952ddc3-f129-4d1e-8d43-c73e01cf8258" (UID: "0952ddc3-f129-4d1e-8d43-c73e01cf8258"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:01.169862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.168692 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-maas-api-tls" pod "0952ddc3-f129-4d1e-8d43-c73e01cf8258" (UID: "0952ddc3-f129-4d1e-8d43-c73e01cf8258"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:01.236976 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.236947 6389 generic.go:356] "Generic (PLEG): container finished" podID="0952ddc3-f129-4d1e-8d43-c73e01cf8258" containerID="8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3" exitCode=0 Nov 03 14:24:01.237134 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.236989 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" Nov 03 14:24:01.237134 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.237029 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" event={"ID":"0952ddc3-f129-4d1e-8d43-c73e01cf8258","Type":"ContainerDied","Data":"8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3"} Nov 03 14:24:01.237134 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.237083 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" event={"ID":"0952ddc3-f129-4d1e-8d43-c73e01cf8258","Type":"ContainerDied","Data":"00b2d5710f5003d5d8dc86ce42906b1dbef88549c2ae82297492018b40068d63"} Nov 03 14:24:01.237134 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.237107 6389 scope.go:122] "RemoveContainer" containerID="8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3" Nov 03 14:24:01.267647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.265943 6389 scope.go:122] "RemoveContainer" containerID="8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3" Nov 03 14:24:01.267647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.267243 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:01.267647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.267268 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vf7c\" (UniqueName: \"kubernetes.io/projected/0952ddc3-f129-4d1e-8d43-c73e01cf8258-kube-api-access-4vf7c\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:01.267647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.267281 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0952ddc3-f129-4d1e-8d43-c73e01cf8258-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:01.267647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.267296 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0952ddc3-f129-4d1e-8d43-c73e01cf8258-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:01.267647 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:24:01.267385 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3\": container with ID starting with 8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3 not found: ID does not exist" containerID="8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3" Nov 03 14:24:01.267647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.267422 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3"} err="failed to get container status \"8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3\": rpc error: code = NotFound desc = could not find container \"8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3\": container with ID starting with 8e37e8adaaca87d3bafff041e4a2b26e47f15a9e951d3df39371f8aab478b4a3 not found: ID does not exist" Nov 03 14:24:01.315693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.315461 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl"] Nov 03 14:24:01.335330 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:01.335301 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl"] Nov 03 14:24:02.584523 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.584489 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr"] Nov 03 14:24:02.585869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.585827 6389 cpu_manager.go:414] "containerMap: removing container" podUID="0952ddc3-f129-4d1e-8d43-c73e01cf8258" containerName="maas-api" Nov 03 14:24:02.585869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.585857 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0952ddc3-f129-4d1e-8d43-c73e01cf8258" containerName="maas-api" Nov 03 14:24:02.586071 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.586021 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="0952ddc3-f129-4d1e-8d43-c73e01cf8258" containerName="maas-api" Nov 03 14:24:02.592937 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.592909 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" Nov 03 14:24:02.646509 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.637400 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e759e5\"" Nov 03 14:24:02.646509 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.637554 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e759e5-dockercfg-5mxhh\"" Nov 03 14:24:02.691665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.684953 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh8q5\" (UniqueName: \"kubernetes.io/projected/6c04787d-8f27-4b2e-bc09-1066c7d38920-kube-api-access-jh8q5\") pod \"payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr\" (UID: \"6c04787d-8f27-4b2e-bc09-1066c7d38920\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" Nov 03 14:24:02.691665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.685193 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c04787d-8f27-4b2e-bc09-1066c7d38920-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr\" (UID: \"6c04787d-8f27-4b2e-bc09-1066c7d38920\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" Nov 03 14:24:02.730489 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.725428 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0952ddc3-f129-4d1e-8d43-c73e01cf8258" path="/var/lib/kubelet/pods/0952ddc3-f129-4d1e-8d43-c73e01cf8258/volumes" Nov 03 14:24:02.730489 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.727950 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr"] Nov 03 14:24:02.786163 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.786126 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c04787d-8f27-4b2e-bc09-1066c7d38920-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr\" (UID: \"6c04787d-8f27-4b2e-bc09-1066c7d38920\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" Nov 03 14:24:02.786329 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.786233 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8q5\" (UniqueName: \"kubernetes.io/projected/6c04787d-8f27-4b2e-bc09-1066c7d38920-kube-api-access-jh8q5\") pod \"payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr\" (UID: \"6c04787d-8f27-4b2e-bc09-1066c7d38920\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" Nov 03 14:24:02.786880 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.786858 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c04787d-8f27-4b2e-bc09-1066c7d38920-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr\" (UID: \"6c04787d-8f27-4b2e-bc09-1066c7d38920\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" Nov 03 14:24:02.806631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:02.806577 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8q5\" (UniqueName: \"kubernetes.io/projected/6c04787d-8f27-4b2e-bc09-1066c7d38920-kube-api-access-jh8q5\") pod \"payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr\" (UID: \"6c04787d-8f27-4b2e-bc09-1066c7d38920\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" Nov 03 14:24:03.015718 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:03.015683 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" Nov 03 14:24:03.317631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:03.317588 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr"] Nov 03 14:24:03.319100 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:24:03.319074 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c04787d_8f27_4b2e_bc09_1066c7d38920.slice/crio-f7fced8c10b1c540295239791556b85461625a9dfc4c34cf2c4e9f02d56d1dcf WatchSource:0}: Error finding container f7fced8c10b1c540295239791556b85461625a9dfc4c34cf2c4e9f02d56d1dcf: Status 404 returned error can't find the container with id f7fced8c10b1c540295239791556b85461625a9dfc4c34cf2c4e9f02d56d1dcf Nov 03 14:24:03.973666 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:03.973578 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4429aa-5fbdf4cc77-6vmjl" podUID="0952ddc3-f129-4d1e-8d43-c73e01cf8258" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.45:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:24:04.255652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:04.255537 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" event={"ID":"6c04787d-8f27-4b2e-bc09-1066c7d38920","Type":"ContainerStarted","Data":"6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24"} Nov 03 14:24:04.255652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:04.255595 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" event={"ID":"6c04787d-8f27-4b2e-bc09-1066c7d38920","Type":"ContainerStarted","Data":"f7fced8c10b1c540295239791556b85461625a9dfc4c34cf2c4e9f02d56d1dcf"} Nov 03 14:24:04.255652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:04.255650 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" Nov 03 14:24:04.297699 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:04.297640 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" podStartSLOduration=2.297627185 podStartE2EDuration="2.297627185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:04.295232986 +0000 UTC m=+719.853869472" watchObservedRunningTime="2026-11-03 14:24:04.297627185 +0000 UTC m=+719.856263694" Nov 03 14:24:10.268354 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.268317 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2"] Nov 03 14:24:10.273796 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.273766 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.296981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.296944 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.297152 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.296995 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zdw\" (UniqueName: \"kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-kube-api-access-r2zdw\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.297152 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.297135 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.297275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.297247 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.297329 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.297290 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.297329 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.297320 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.297466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.297346 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.297466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.297396 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.297466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.297425 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.302336 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.302309 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2"] Nov 03 14:24:10.398095 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398062 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.398291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398146 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.398291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398190 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.398291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398220 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.398291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398246 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.398530 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398296 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.398530 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398324 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.398530 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398451 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.398530 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398481 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zdw\" (UniqueName: \"kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-kube-api-access-r2zdw\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.398753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398599 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.398753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398672 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.398876 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398847 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.398931 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.398919 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.399352 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.399316 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.400971 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.400948 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.401086 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.401013 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.421085 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.421057 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.421359 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.421337 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zdw\" (UniqueName: \"kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-kube-api-access-r2zdw\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.590872 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.590794 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:10.917146 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.917114 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2"] Nov 03 14:24:10.919414 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:24:10.919377 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0cd10d_efc2_47f3_860c_5db6ed4b38db.slice/crio-222c06d7b5c9c65685f564a7d4652c8ff43bbb2233af1b7ac73546c105a186f1 WatchSource:0}: Error finding container 222c06d7b5c9c65685f564a7d4652c8ff43bbb2233af1b7ac73546c105a186f1: Status 404 returned error can't find the container with id 222c06d7b5c9c65685f564a7d4652c8ff43bbb2233af1b7ac73546c105a186f1 Nov 03 14:24:10.921763 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.921735 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:24:10.921840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.921796 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:24:10.921840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:10.921824 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:24:11.304846 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:11.304811 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" event={"ID":"3d0cd10d-efc2-47f3-860c-5db6ed4b38db","Type":"ContainerStarted","Data":"6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b"} Nov 03 14:24:11.305247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:11.304860 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" event={"ID":"3d0cd10d-efc2-47f3-860c-5db6ed4b38db","Type":"ContainerStarted","Data":"222c06d7b5c9c65685f564a7d4652c8ff43bbb2233af1b7ac73546c105a186f1"} Nov 03 14:24:11.591080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:11.590997 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:11.592855 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:11.592829 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 14:24:11.592984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:11.592875 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 14:24:12.591571 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:12.591535 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 14:24:12.592015 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:12.591592 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 14:24:13.592104 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:13.592005 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 14:24:13.592104 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:13.592080 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 14:24:13.652635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:13.641224 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" podStartSLOduration=3.641205511 podStartE2EDuration="3.641205511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:11.375091684 +0000 UTC m=+726.933728189" watchObservedRunningTime="2026-11-03 14:24:13.641205511 +0000 UTC m=+729.199841996" Nov 03 14:24:13.654580 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:13.654533 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g"] Nov 03 14:24:13.654894 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:13.654867 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" podUID="4d691739-5007-41c9-acf9-445e68d9561f" containerName="istio-proxy" containerID="cri-o://b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755" gracePeriod=30 Nov 03 14:24:14.591374 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:14.591342 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 14:24:14.591563 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:14.591397 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 14:24:15.264522 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:15.264495 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" Nov 03 14:24:15.591177 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:15.591101 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 14:24:15.591177 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:15.591158 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 14:24:16.474618 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.474582 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg"] Nov 03 14:24:16.478804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.478782 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:16.498351 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.498328 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg"] Nov 03 14:24:16.507372 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.507338 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bpc5\" (UniqueName: \"kubernetes.io/projected/e6180463-5ac5-4422-9b30-d60723c15deb-kube-api-access-4bpc5\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:16.507577 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.507380 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-metrics-certs\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:16.507577 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.507467 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6180463-5ac5-4422-9b30-d60723c15deb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:16.507577 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.507549 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-maas-api-tls\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:16.508322 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.508298 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e759e5-serving-cert\"" Nov 03 14:24:16.591828 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.591795 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 14:24:16.591990 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.591850 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 14:24:16.608518 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.608488 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bpc5\" (UniqueName: \"kubernetes.io/projected/e6180463-5ac5-4422-9b30-d60723c15deb-kube-api-access-4bpc5\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:16.608698 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.608538 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-metrics-certs\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:16.608698 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.608637 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6180463-5ac5-4422-9b30-d60723c15deb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:16.608809 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.608702 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-maas-api-tls\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:16.608849 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:24:16.608819 6389 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-serving-cert: secret "maas-api-e2e-inf-b-e759e5-serving-cert" not found Nov 03 14:24:16.608890 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:24:16.608877 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-maas-api-tls podName:e6180463-5ac5-4422-9b30-d60723c15deb nodeName:}" failed. No retries permitted until 2026-11-03 14:24:17.108856236 +0000 UTC m=+732.667492701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-maas-api-tls") pod "maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" (UID: "e6180463-5ac5-4422-9b30-d60723c15deb") : secret "maas-api-e2e-inf-b-e759e5-serving-cert" not found Nov 03 14:24:16.609315 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.609293 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6180463-5ac5-4422-9b30-d60723c15deb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:16.611569 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.611546 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-metrics-certs\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:16.653627 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:16.653402 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bpc5\" (UniqueName: \"kubernetes.io/projected/e6180463-5ac5-4422-9b30-d60723c15deb-kube-api-access-4bpc5\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:17.115084 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:17.115051 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-maas-api-tls\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:17.117573 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:17.117545 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-maas-api-tls\") pod \"maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:17.393311 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:17.393229 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:17.698971 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:17.698940 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg"] Nov 03 14:24:17.700581 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:24:17.700551 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6180463_5ac5_4422_9b30_d60723c15deb.slice/crio-d93f804cd389b5f54880cebaeec401ea611211e705cb5b6b50de0c81e28b8def WatchSource:0}: Error finding container d93f804cd389b5f54880cebaeec401ea611211e705cb5b6b50de0c81e28b8def: Status 404 returned error can't find the container with id d93f804cd389b5f54880cebaeec401ea611211e705cb5b6b50de0c81e28b8def Nov 03 14:24:18.020511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:18.020429 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:18.020847 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:18.020825 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:18.021634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:18.021584 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:24:18.398991 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:18.398900 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" event={"ID":"e6180463-5ac5-4422-9b30-d60723c15deb","Type":"ContainerStarted","Data":"5592f41795ed9cf6d8d82d276dd5a8a68692ba3fc7d2c2b40516b496c8e788dc"} Nov 03 14:24:18.399273 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:18.399242 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" event={"ID":"e6180463-5ac5-4422-9b30-d60723c15deb","Type":"ContainerStarted","Data":"d93f804cd389b5f54880cebaeec401ea611211e705cb5b6b50de0c81e28b8def"} Nov 03 14:24:18.448632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:18.448317 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" podStartSLOduration=1.9587960039999999 podStartE2EDuration="2.448295771s" totalImagesPullingTime="489.499767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:18.445067348 +0000 UTC m=+734.003703832" watchObservedRunningTime="2026-11-03 14:24:18.448295771 +0000 UTC m=+734.006932259" Nov 03 14:24:18.977326 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:18.977242 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.159984 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-data\") pod \"4d691739-5007-41c9-acf9-445e68d9561f\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.160119 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-credential-socket\") pod \"4d691739-5007-41c9-acf9-445e68d9561f\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.160160 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d691739-5007-41c9-acf9-445e68d9561f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d691739-5007-41c9-acf9-445e68d9561f-istio-podinfo\") pod \"4d691739-5007-41c9-acf9-445e68d9561f\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.160207 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-istio-token\" (UniqueName: \"kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-istio-token\") pod \"4d691739-5007-41c9-acf9-445e68d9561f\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.160253 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d691739-5007-41c9-acf9-445e68d9561f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d691739-5007-41c9-acf9-445e68d9561f-istiod-ca-cert\") pod \"4d691739-5007-41c9-acf9-445e68d9561f\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.160596 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-certs\") pod \"4d691739-5007-41c9-acf9-445e68d9561f\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.160746 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-kube-api-access-p2wk2\" (UniqueName: \"kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-kube-api-access-p2wk2\") pod \"4d691739-5007-41c9-acf9-445e68d9561f\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.160790 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-envoy\") pod \"4d691739-5007-41c9-acf9-445e68d9561f\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.160852 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-socket\") pod \"4d691739-5007-41c9-acf9-445e68d9561f\" (UID: \"4d691739-5007-41c9-acf9-445e68d9561f\") " Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.161095 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-credential-socket" pod "4d691739-5007-41c9-acf9-445e68d9561f" (UID: "4d691739-5007-41c9-acf9-445e68d9561f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.161316 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.161409 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-data" pod "4d691739-5007-41c9-acf9-445e68d9561f" (UID: "4d691739-5007-41c9-acf9-445e68d9561f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.161583 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-socket" pod "4d691739-5007-41c9-acf9-445e68d9561f" (UID: "4d691739-5007-41c9-acf9-445e68d9561f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:19.161638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.161638 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d691739-5007-41c9-acf9-445e68d9561f-istiod-ca-cert" pod "4d691739-5007-41c9-acf9-445e68d9561f" (UID: "4d691739-5007-41c9-acf9-445e68d9561f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:19.162905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.162881 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-certs" pod "4d691739-5007-41c9-acf9-445e68d9561f" (UID: "4d691739-5007-41c9-acf9-445e68d9561f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:19.166412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.166375 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-kube-api-access-p2wk2" pod "4d691739-5007-41c9-acf9-445e68d9561f" (UID: "4d691739-5007-41c9-acf9-445e68d9561f"). InnerVolumeSpecName "kube-api-access-p2wk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:19.169360 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.169327 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-envoy" pod "4d691739-5007-41c9-acf9-445e68d9561f" (UID: "4d691739-5007-41c9-acf9-445e68d9561f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:19.169658 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.169584 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-istio-token" pod "4d691739-5007-41c9-acf9-445e68d9561f" (UID: "4d691739-5007-41c9-acf9-445e68d9561f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:19.169849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.169811 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d691739-5007-41c9-acf9-445e68d9561f-istio-podinfo" pod "4d691739-5007-41c9-acf9-445e68d9561f" (UID: "4d691739-5007-41c9-acf9-445e68d9561f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:24:19.263057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.262952 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:19.263057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.262992 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d691739-5007-41c9-acf9-445e68d9561f-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:19.263057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.263008 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:19.263057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.263022 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d691739-5007-41c9-acf9-445e68d9561f-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:19.263057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.263038 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:19.263057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.263052 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2wk2\" (UniqueName: \"kubernetes.io/projected/4d691739-5007-41c9-acf9-445e68d9561f-kube-api-access-p2wk2\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:19.263539 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.263068 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:19.263539 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.263082 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d691739-5007-41c9-acf9-445e68d9561f-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:19.405858 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.405821 6389 generic.go:356] "Generic (PLEG): container finished" podID="4d691739-5007-41c9-acf9-445e68d9561f" containerID="b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755" exitCode=0 Nov 03 14:24:19.406054 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.405873 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" Nov 03 14:24:19.406054 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.405901 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" event={"ID":"4d691739-5007-41c9-acf9-445e68d9561f","Type":"ContainerDied","Data":"b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755"} Nov 03 14:24:19.406054 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.405957 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g" event={"ID":"4d691739-5007-41c9-acf9-445e68d9561f","Type":"ContainerDied","Data":"fa3a7fdcf882dde5077215d687830fba3868451161bb6c76afcf341e9eebb349"} Nov 03 14:24:19.406054 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.405978 6389 scope.go:122] "RemoveContainer" containerID="b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755" Nov 03 14:24:19.406415 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.406389 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:19.416499 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.416481 6389 scope.go:122] "RemoveContainer" containerID="82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce" Nov 03 14:24:19.433024 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.432910 6389 scope.go:122] "RemoveContainer" containerID="b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755" Nov 03 14:24:19.433644 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:24:19.433493 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755\": container with ID starting with b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755 not found: ID does not exist" containerID="b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755" Nov 03 14:24:19.433644 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.433527 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755"} err="failed to get container status \"b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755\": rpc error: code = NotFound desc = could not find container \"b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755\": container with ID starting with b9c13144b2eb0d75827d9822ccea39abfa9a1cf6e7dce40722631378436fc755 not found: ID does not exist" Nov 03 14:24:19.433644 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.433550 6389 scope.go:122] "RemoveContainer" containerID="82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce" Nov 03 14:24:19.434234 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:24:19.434165 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce\": container with ID starting with 82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce not found: ID does not exist" containerID="82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce" Nov 03 14:24:19.434234 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.434196 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce"} err="failed to get container status \"82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce\": rpc error: code = NotFound desc = could not find container \"82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce\": container with ID starting with 82cdfa9cdcfc74dec57a0a477e7000dcc6854011cd52d05e390966d2f9ef94ce not found: ID does not exist" Nov 03 14:24:19.478714 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.477869 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g"] Nov 03 14:24:19.485932 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:19.485900 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4429aa-openshift-default-856555d6d5-8g87g"] Nov 03 14:24:20.619961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:20.619919 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d691739-5007-41c9-acf9-445e68d9561f" path="/var/lib/kubelet/pods/4d691739-5007-41c9-acf9-445e68d9561f/volumes" Nov 03 14:24:24.626101 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:24.626066 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm"] Nov 03 14:24:25.420446 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:25.420420 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:24:28.441670 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:24:28.438187 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:24:29.634636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:29.633296 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" podUID="cd7d2837-bbab-4c2a-acdc-f259a2e80e30" containerName="payload-pre-processing" containerID="cri-o://3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382" gracePeriod=25 Nov 03 14:24:30.303970 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.303824 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" Nov 03 14:24:30.310506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.310477 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-plugins-config-volume\") pod \"cd7d2837-bbab-4c2a-acdc-f259a2e80e30\" (UID: \"cd7d2837-bbab-4c2a-acdc-f259a2e80e30\") " Nov 03 14:24:30.310700 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.310563 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-kube-api-access-ndmc4\" (UniqueName: \"kubernetes.io/projected/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-kube-api-access-ndmc4\") pod \"cd7d2837-bbab-4c2a-acdc-f259a2e80e30\" (UID: \"cd7d2837-bbab-4c2a-acdc-f259a2e80e30\") " Nov 03 14:24:30.310867 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.310843 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-plugins-config-volume" pod "cd7d2837-bbab-4c2a-acdc-f259a2e80e30" (UID: "cd7d2837-bbab-4c2a-acdc-f259a2e80e30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:30.311325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.311307 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:30.312857 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.312830 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-kube-api-access-ndmc4" pod "cd7d2837-bbab-4c2a-acdc-f259a2e80e30" (UID: "cd7d2837-bbab-4c2a-acdc-f259a2e80e30"). InnerVolumeSpecName "kube-api-access-ndmc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:30.413304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.413269 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndmc4\" (UniqueName: \"kubernetes.io/projected/cd7d2837-bbab-4c2a-acdc-f259a2e80e30-kube-api-access-ndmc4\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:30.490966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.490871 6389 generic.go:356] "Generic (PLEG): container finished" podID="cd7d2837-bbab-4c2a-acdc-f259a2e80e30" containerID="3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382" exitCode=0 Nov 03 14:24:30.490966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.490915 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" event={"ID":"cd7d2837-bbab-4c2a-acdc-f259a2e80e30","Type":"ContainerDied","Data":"3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382"} Nov 03 14:24:30.490966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.490957 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" event={"ID":"cd7d2837-bbab-4c2a-acdc-f259a2e80e30","Type":"ContainerDied","Data":"a76c93b65743793daa25bafdd24bdad8502b8bee9440271121ba6be047a6cd8f"} Nov 03 14:24:30.491266 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.490977 6389 scope.go:122] "RemoveContainer" containerID="3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382" Nov 03 14:24:30.491266 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.491097 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" Nov 03 14:24:30.503540 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.503516 6389 scope.go:122] "RemoveContainer" containerID="3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382" Nov 03 14:24:30.503863 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:24:30.503842 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382\": container with ID starting with 3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382 not found: ID does not exist" containerID="3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382" Nov 03 14:24:30.503937 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.503873 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382"} err="failed to get container status \"3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382\": rpc error: code = NotFound desc = could not find container \"3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382\": container with ID starting with 3360d1e5e80bdf45e2785c5c38da59b7b25ace4962a00584d2ad2212d4f87382 not found: ID does not exist" Nov 03 14:24:30.546638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.546590 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm"] Nov 03 14:24:30.554778 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.554737 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm"] Nov 03 14:24:30.618063 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:30.618023 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd7d2837-bbab-4c2a-acdc-f259a2e80e30" path="/var/lib/kubelet/pods/cd7d2837-bbab-4c2a-acdc-f259a2e80e30/volumes" Nov 03 14:24:31.192560 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:31.192516 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0a0ec6-fcfcfc679-nfbhm" podUID="cd7d2837-bbab-4c2a-acdc-f259a2e80e30" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.48:50052\" within 1s: context deadline exceeded" Nov 03 14:24:31.192998 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:24:31.192633 6389 logging.go:55] [core] [Channel #443 SubChannel #444]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.48:50052", ServerName: "10.134.0.48:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.48:50052: operation was canceled" Nov 03 14:24:37.738928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:37.738882 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65"] Nov 03 14:24:37.740731 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:37.740681 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" podUID="846a80aa-400e-4062-a2b5-057ea0124554" containerName="maas-api" containerID="cri-o://19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11" gracePeriod=30 Nov 03 14:24:38.395634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.391810 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:24:38.578688 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.575891 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846a80aa-400e-4062-a2b5-057ea0124554-kube-api-access-7cdkd\" (UniqueName: \"kubernetes.io/projected/846a80aa-400e-4062-a2b5-057ea0124554-kube-api-access-7cdkd\") pod \"846a80aa-400e-4062-a2b5-057ea0124554\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " Nov 03 14:24:38.578688 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.576091 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-maas-api-tls\") pod \"846a80aa-400e-4062-a2b5-057ea0124554\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " Nov 03 14:24:38.578688 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.576130 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-metrics-certs\" (UniqueName: \"kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-metrics-certs\") pod \"846a80aa-400e-4062-a2b5-057ea0124554\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " Nov 03 14:24:38.578688 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.576196 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846a80aa-400e-4062-a2b5-057ea0124554-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846a80aa-400e-4062-a2b5-057ea0124554-openshift-service-ca\") pod \"846a80aa-400e-4062-a2b5-057ea0124554\" (UID: \"846a80aa-400e-4062-a2b5-057ea0124554\") " Nov 03 14:24:38.590991 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.590929 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846a80aa-400e-4062-a2b5-057ea0124554-openshift-service-ca" pod "846a80aa-400e-4062-a2b5-057ea0124554" (UID: "846a80aa-400e-4062-a2b5-057ea0124554"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:38.608339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.608290 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-metrics-certs" pod "846a80aa-400e-4062-a2b5-057ea0124554" (UID: "846a80aa-400e-4062-a2b5-057ea0124554"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:38.608747 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.608700 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-maas-api-tls" pod "846a80aa-400e-4062-a2b5-057ea0124554" (UID: "846a80aa-400e-4062-a2b5-057ea0124554"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:38.613920 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.613881 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846a80aa-400e-4062-a2b5-057ea0124554-kube-api-access-7cdkd" pod "846a80aa-400e-4062-a2b5-057ea0124554" (UID: "846a80aa-400e-4062-a2b5-057ea0124554"). InnerVolumeSpecName "kube-api-access-7cdkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:38.616068 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.616021 6389 generic.go:356] "Generic (PLEG): container finished" podID="846a80aa-400e-4062-a2b5-057ea0124554" containerID="19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11" exitCode=0 Nov 03 14:24:38.616342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.616280 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" Nov 03 14:24:38.701685 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.701642 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" event={"ID":"846a80aa-400e-4062-a2b5-057ea0124554","Type":"ContainerDied","Data":"19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11"} Nov 03 14:24:38.701953 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.701938 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65" event={"ID":"846a80aa-400e-4062-a2b5-057ea0124554","Type":"ContainerDied","Data":"9e76d7bad60c9389ca4bc0ee2849fe627b2f1f54f55b87a030b2a2b73dd0d811"} Nov 03 14:24:38.702063 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.702052 6389 scope.go:122] "RemoveContainer" containerID="19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11" Nov 03 14:24:38.707637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.707452 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cdkd\" (UniqueName: \"kubernetes.io/projected/846a80aa-400e-4062-a2b5-057ea0124554-kube-api-access-7cdkd\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:38.707637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.707486 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:38.707637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.707500 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846a80aa-400e-4062-a2b5-057ea0124554-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:38.707637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.707514 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846a80aa-400e-4062-a2b5-057ea0124554-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:24:38.740790 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.740763 6389 scope.go:122] "RemoveContainer" containerID="19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11" Nov 03 14:24:38.743017 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:24:38.742985 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11\": container with ID starting with 19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11 not found: ID does not exist" containerID="19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11" Nov 03 14:24:38.743186 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.743158 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11"} err="failed to get container status \"19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11\": rpc error: code = NotFound desc = could not find container \"19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11\": container with ID starting with 19af969d7a5c73222b8b2464681ff5cf3cd664cd392630b58c29d612f046bd11 not found: ID does not exist" Nov 03 14:24:38.788415 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.788384 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65"] Nov 03 14:24:38.798124 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:38.798097 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0a0ec6-6f5875ffc6-5wx65"] Nov 03 14:24:40.618853 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:40.618821 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846a80aa-400e-4062-a2b5-057ea0124554" path="/var/lib/kubelet/pods/846a80aa-400e-4062-a2b5-057ea0124554/volumes" Nov 03 14:24:43.052795 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.052764 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb"] Nov 03 14:24:43.053282 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.053269 6389 cpu_manager.go:414] "containerMap: removing container" podUID="cd7d2837-bbab-4c2a-acdc-f259a2e80e30" containerName="payload-pre-processing" Nov 03 14:24:43.053282 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.053279 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7d2837-bbab-4c2a-acdc-f259a2e80e30" containerName="payload-pre-processing" Nov 03 14:24:43.053420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.053293 6389 cpu_manager.go:414] "containerMap: removing container" podUID="4d691739-5007-41c9-acf9-445e68d9561f" containerName="istio-proxy" Nov 03 14:24:43.053420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.053299 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d691739-5007-41c9-acf9-445e68d9561f" containerName="istio-proxy" Nov 03 14:24:43.053420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.053310 6389 cpu_manager.go:414] "containerMap: removing container" podUID="846a80aa-400e-4062-a2b5-057ea0124554" containerName="maas-api" Nov 03 14:24:43.053420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.053315 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846a80aa-400e-4062-a2b5-057ea0124554" containerName="maas-api" Nov 03 14:24:43.053420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.053408 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d691739-5007-41c9-acf9-445e68d9561f" containerName="istio-proxy" Nov 03 14:24:43.053420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.053415 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7d2837-bbab-4c2a-acdc-f259a2e80e30" containerName="payload-pre-processing" Nov 03 14:24:43.053420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.053422 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d691739-5007-41c9-acf9-445e68d9561f" containerName="istio-proxy" Nov 03 14:24:43.053794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.053429 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="846a80aa-400e-4062-a2b5-057ea0124554" containerName="maas-api" Nov 03 14:24:43.058407 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.058389 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.085468 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.085439 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5b97f1\"/\"openshift-service-ca.crt\"" Nov 03 14:24:43.085468 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.085478 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5b97f1\"/\"test-model-5b97f1-kserve-self-signed-certs\"" Nov 03 14:24:43.085716 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.085499 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5b97f1\"/\"kube-root-ca.crt\"" Nov 03 14:24:43.085716 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.085567 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5b97f1\"/\"default-dockercfg-ks22n\"" Nov 03 14:24:43.087821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.087797 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb"] Nov 03 14:24:43.171166 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.171129 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-home\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.171330 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.171221 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-dshm\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.171330 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.171318 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tls-certs\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.171470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.171414 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-model-cache\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.171520 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.171510 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tmp-dir\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.171708 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.171691 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f65h\" (UniqueName: \"kubernetes.io/projected/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-kube-api-access-8f65h\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.272979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.272942 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-model-cache\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.273144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.273006 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tmp-dir\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.273144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.273110 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f65h\" (UniqueName: \"kubernetes.io/projected/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-kube-api-access-8f65h\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.273250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.273148 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-home\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.273250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.273218 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-dshm\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.273250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.273247 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tls-certs\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.273408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.273390 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-model-cache\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.273500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.273480 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tmp-dir\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.273598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.273580 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-home\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.275758 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.275730 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-dshm\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.275918 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.275900 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tls-certs\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.302773 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.302741 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f65h\" (UniqueName: \"kubernetes.io/projected/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-kube-api-access-8f65h\") pod \"test-model-5b97f1-kserve-84558985dc-c6qfb\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.379765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.379666 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:43.731295 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:43.731268 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb"] Nov 03 14:24:43.732104 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:24:43.732075 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8cd3ee4_c080_4cd5_a6bb_4524d0ac91d8.slice/crio-07f9deed731f1e0dcc57016fd80e48d18d48a9a80e939a0116af8ad0727c02ba WatchSource:0}: Error finding container 07f9deed731f1e0dcc57016fd80e48d18d48a9a80e939a0116af8ad0727c02ba: Status 404 returned error can't find the container with id 07f9deed731f1e0dcc57016fd80e48d18d48a9a80e939a0116af8ad0727c02ba Nov 03 14:24:44.663309 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:44.663270 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" event={"ID":"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8","Type":"ContainerStarted","Data":"1296d6f0cbc41e432a2486ce51a5a123221958070e82265a8bb6b5981d785a38"} Nov 03 14:24:44.663309 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:44.663317 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" event={"ID":"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8","Type":"ContainerStarted","Data":"07f9deed731f1e0dcc57016fd80e48d18d48a9a80e939a0116af8ad0727c02ba"} Nov 03 14:24:44.704663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:44.704600 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" podStartSLOduration=1.704587175 podStartE2EDuration="1.704587175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:44.704069583 +0000 UTC m=+760.262706069" watchObservedRunningTime="2026-11-03 14:24:44.704587175 +0000 UTC m=+760.263223658" Nov 03 14:24:45.380372 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.380334 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:45.742943 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.742913 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr"] Nov 03 14:24:45.743766 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.743747 6389 cpu_manager.go:414] "containerMap: removing container" podUID="4d691739-5007-41c9-acf9-445e68d9561f" containerName="istio-proxy" Nov 03 14:24:45.743766 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.743766 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d691739-5007-41c9-acf9-445e68d9561f" containerName="istio-proxy" Nov 03 14:24:45.747308 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.747286 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.763165 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.763130 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-331c9e-openshift-default-dockercfg-skzpv\"" Nov 03 14:24:45.776450 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.776424 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr"] Nov 03 14:24:45.812594 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.812569 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.812753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.812627 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fccn\" (UniqueName: \"kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-kube-api-access-8fccn\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.812753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.812715 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-token\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.812849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.812792 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-podinfo\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.812974 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.812954 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-credential-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.813030 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.812982 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-data\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.813030 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.813005 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-certs\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.813185 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.813163 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e41d26-58b4-42ce-a010-e2fa486614d1-istiod-ca-cert\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.813249 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.813226 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-envoy\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.914312 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-credential-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.914365 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-data\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.914396 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-certs\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.914466 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e41d26-58b4-42ce-a010-e2fa486614d1-istiod-ca-cert\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.914513 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-envoy\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.914557 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.914591 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fccn\" (UniqueName: \"kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-kube-api-access-8fccn\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.914647 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-token\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.914719 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-podinfo\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.915990 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e41d26-58b4-42ce-a010-e2fa486614d1-istiod-ca-cert\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.916253 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-credential-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.916471 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-data\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.916719 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-certs\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.917635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.916918 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.940522 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.940489 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fccn\" (UniqueName: \"kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-kube-api-access-8fccn\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.941385 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.941256 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-token\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.971283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.971233 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-envoy\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:45.971283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:45.971232 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-podinfo\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:46.063042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:46.062959 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:46.379761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:46.379733 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr"] Nov 03 14:24:46.381189 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:24:46.381158 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e41d26_58b4_42ce_a010_e2fa486614d1.slice/crio-3a5ecdeb00849d0d6f9aadf04a6fdd2ee37ed6cea5297d3a4c72a4eeab17da08 WatchSource:0}: Error finding container 3a5ecdeb00849d0d6f9aadf04a6fdd2ee37ed6cea5297d3a4c72a4eeab17da08: Status 404 returned error can't find the container with id 3a5ecdeb00849d0d6f9aadf04a6fdd2ee37ed6cea5297d3a4c72a4eeab17da08 Nov 03 14:24:46.383746 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:46.383711 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:24:46.383875 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:46.383790 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:24:46.383875 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:46.383851 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:24:46.677672 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:46.677404 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" event={"ID":"a1e41d26-58b4-42ce-a010-e2fa486614d1","Type":"ContainerStarted","Data":"88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563"} Nov 03 14:24:46.677672 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:46.677474 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" event={"ID":"a1e41d26-58b4-42ce-a010-e2fa486614d1","Type":"ContainerStarted","Data":"3a5ecdeb00849d0d6f9aadf04a6fdd2ee37ed6cea5297d3a4c72a4eeab17da08"} Nov 03 14:24:46.721458 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:46.721408 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" podStartSLOduration=1.72139412 podStartE2EDuration="1.72139412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:46.719178392 +0000 UTC m=+762.277814873" watchObservedRunningTime="2026-11-03 14:24:46.72139412 +0000 UTC m=+762.280030604" Nov 03 14:24:47.063661 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:47.063573 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:47.069282 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:47.069250 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:47.682869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:47.682836 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:47.683945 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:47.683924 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:24:53.379875 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:53.379838 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:53.393387 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:53.393360 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:53.797889 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:53.797857 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:24:58.087015 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.086973 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh"] Nov 03 14:24:58.091177 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.091159 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.119433 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.119403 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e759e5\"/\"openshift-service-ca.crt\"" Nov 03 14:24:58.119433 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.119427 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e759e5\"/\"test-model-e759e5-kserve-self-signed-certs\"" Nov 03 14:24:58.119433 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.119411 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e759e5\"/\"kube-root-ca.crt\"" Nov 03 14:24:58.119672 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.119475 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e759e5\"/\"default-dockercfg-sx9v8\"" Nov 03 14:24:58.123104 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.123080 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh"] Nov 03 14:24:58.153217 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.153189 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnql\" (UniqueName: \"kubernetes.io/projected/20313390-9803-4c52-898e-23ce0fed0365-kube-api-access-dmnql\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.153373 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.153228 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-home\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.153373 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.153311 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-dshm\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.153460 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.153433 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-model-cache\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.153506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.153478 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20313390-9803-4c52-898e-23ce0fed0365-tls-certs\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.153597 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.153579 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-tmp-dir\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.254948 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.254916 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnql\" (UniqueName: \"kubernetes.io/projected/20313390-9803-4c52-898e-23ce0fed0365-kube-api-access-dmnql\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.254948 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.254953 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-home\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.255182 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.254979 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-dshm\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.255182 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.255059 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-model-cache\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.255182 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.255096 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20313390-9803-4c52-898e-23ce0fed0365-tls-certs\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.255182 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.255124 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-tmp-dir\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.255402 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.255379 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-home\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.255465 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.255434 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-tmp-dir\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.255537 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.255517 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-model-cache\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.257314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.257298 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-dshm\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.257717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.257687 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20313390-9803-4c52-898e-23ce0fed0365-tls-certs\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.275432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.275402 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnql\" (UniqueName: \"kubernetes.io/projected/20313390-9803-4c52-898e-23ce0fed0365-kube-api-access-dmnql\") pod \"test-model-e759e5-kserve-584d466575-r5grh\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.406678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.405855 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:24:58.436747 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:24:58.436710 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:24:58.717682 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.717648 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh"] Nov 03 14:24:58.718202 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:24:58.718165 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20313390_9803_4c52_898e_23ce0fed0365.slice/crio-14e59a6d451a7f8c5de99e26707cc57ae31d59627444c8bd38e2c6611a430b62 WatchSource:0}: Error finding container 14e59a6d451a7f8c5de99e26707cc57ae31d59627444c8bd38e2c6611a430b62: Status 404 returned error can't find the container with id 14e59a6d451a7f8c5de99e26707cc57ae31d59627444c8bd38e2c6611a430b62 Nov 03 14:24:58.819580 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.818959 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" event={"ID":"20313390-9803-4c52-898e-23ce0fed0365","Type":"ContainerStarted","Data":"379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524"} Nov 03 14:24:58.819580 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.819006 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" event={"ID":"20313390-9803-4c52-898e-23ce0fed0365","Type":"ContainerStarted","Data":"14e59a6d451a7f8c5de99e26707cc57ae31d59627444c8bd38e2c6611a430b62"} Nov 03 14:24:58.903803 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:58.903746 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" podStartSLOduration=0.903730641 podStartE2EDuration="903.730641ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:58.901542992 +0000 UTC m=+774.460179477" watchObservedRunningTime="2026-11-03 14:24:58.903730641 +0000 UTC m=+774.462367124" Nov 03 14:24:59.406909 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:24:59.406876 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:25:05.267388 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.267352 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t"] Nov 03 14:25:05.272986 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.272961 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.300745 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.300663 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-331c9e-serving-cert\"" Nov 03 14:25:05.304573 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.304545 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t"] Nov 03 14:25:05.409189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.409152 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-metrics-certs\") pod \"maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.409406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.409228 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.409406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.409352 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.409406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.409391 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8phq\" (UniqueName: \"kubernetes.io/projected/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-kube-api-access-h8phq\") pod \"maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.518676 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.512960 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8phq\" (UniqueName: \"kubernetes.io/projected/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-kube-api-access-h8phq\") pod \"maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.518676 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.513160 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-metrics-certs\") pod \"maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.518676 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.513292 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.518676 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.513366 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.518676 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.514189 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.538177 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.538002 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.538330 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.538284 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-metrics-certs\") pod \"maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.540928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.540901 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8phq\" (UniqueName: \"kubernetes.io/projected/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-kube-api-access-h8phq\") pod \"maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.587702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.587650 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:05.898779 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:05.898756 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t"] Nov 03 14:25:05.899944 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:25:05.899918 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7fe6f4_8d7d_4b1e_b0ec_9963cd72cbc9.slice/crio-ca5cce4b27c928225bae2630f3d7225ccbc6387ae6494e1db90cfefac3a15034 WatchSource:0}: Error finding container ca5cce4b27c928225bae2630f3d7225ccbc6387ae6494e1db90cfefac3a15034: Status 404 returned error can't find the container with id ca5cce4b27c928225bae2630f3d7225ccbc6387ae6494e1db90cfefac3a15034 Nov 03 14:25:06.898024 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:06.897992 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" event={"ID":"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9","Type":"ContainerStarted","Data":"6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b"} Nov 03 14:25:06.898422 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:06.898053 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" event={"ID":"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9","Type":"ContainerStarted","Data":"ca5cce4b27c928225bae2630f3d7225ccbc6387ae6494e1db90cfefac3a15034"} Nov 03 14:25:06.898422 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:06.898079 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:06.951225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:06.951169 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" podStartSLOduration=1.599622256 podStartE2EDuration="1.951155866s" totalImagesPullingTime="351.53361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:06.94898603 +0000 UTC m=+782.507622515" watchObservedRunningTime="2026-11-03 14:25:06.951155866 +0000 UTC m=+782.509792349" Nov 03 14:25:08.407266 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:08.407223 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:25:08.420170 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:08.420143 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:25:08.921236 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:08.921206 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:25:10.457419 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.457380 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4"] Nov 03 14:25:10.462586 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.462560 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.496061 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.496025 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4"] Nov 03 14:25:10.505272 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.505240 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.505524 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.505500 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.505730 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.505705 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.505861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.505758 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.506092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.506021 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.506177 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.506157 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c43724b-ae54-408e-8e4d-b307cd947d9c-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.506338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.506320 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.506432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.506416 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.506504 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.506482 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47cj8\" (UniqueName: \"kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-kube-api-access-47cj8\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.608982 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.608939 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c43724b-ae54-408e-8e4d-b307cd947d9c-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.609309 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.609288 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.609479 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.609462 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.609594 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.609579 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47cj8\" (UniqueName: \"kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-kube-api-access-47cj8\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.609765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.609750 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.609878 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.609865 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.609987 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.609972 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.610104 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.610088 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.610335 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.610318 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.611733 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.611702 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c43724b-ae54-408e-8e4d-b307cd947d9c-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.612030 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.612005 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.612783 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.612756 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.613247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.613223 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.613388 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.613359 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.616875 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.616851 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.617500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.617427 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.655744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.655708 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47cj8\" (UniqueName: \"kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-kube-api-access-47cj8\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.662525 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.662495 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:10.787777 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:10.787686 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:11.249293 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:11.249243 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4"] Nov 03 14:25:11.293362 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:11.293314 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:25:11.293499 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:11.293419 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:25:11.293499 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:11.293464 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:25:11.984991 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:11.984955 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" event={"ID":"5c43724b-ae54-408e-8e4d-b307cd947d9c","Type":"ContainerStarted","Data":"d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da"} Nov 03 14:25:11.985417 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:11.985030 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" event={"ID":"5c43724b-ae54-408e-8e4d-b307cd947d9c","Type":"ContainerStarted","Data":"0ba27f8bc43faeee4021df298133c6a1204139386a95851535fb0a28152999e8"} Nov 03 14:25:12.041040 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:12.040977 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" podStartSLOduration=2.040958274 podStartE2EDuration="2.040958274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:12.039063909 +0000 UTC m=+787.597700488" watchObservedRunningTime="2026-11-03 14:25:12.040958274 +0000 UTC m=+787.599594761" Nov 03 14:25:12.788237 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:12.788203 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:12.790094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:12.790065 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 14:25:12.790217 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:12.790159 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" podUID="5c43724b-ae54-408e-8e4d-b307cd947d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 14:25:12.909971 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:12.909939 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:25:13.789096 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:13.789064 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 14:25:13.789590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:13.789114 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" podUID="5c43724b-ae54-408e-8e4d-b307cd947d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 14:25:14.788994 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:14.788953 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 14:25:14.789190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:14.789019 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" podUID="5c43724b-ae54-408e-8e4d-b307cd947d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 14:25:15.789324 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:15.789289 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 14:25:15.789751 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:15.789349 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" podUID="5c43724b-ae54-408e-8e4d-b307cd947d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 14:25:16.789092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:16.789060 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 14:25:16.789275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:16.789124 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" podUID="5c43724b-ae54-408e-8e4d-b307cd947d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 14:25:17.789201 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:17.789166 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 14:25:17.789638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:17.789228 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" podUID="5c43724b-ae54-408e-8e4d-b307cd947d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 14:25:18.878681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:18.878636 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:18.879544 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:18.879457 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:18.880808 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:18.880785 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:25:24.227744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:24.227696 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" podUID="76e983d7-ca69-44d1-857c-65eb03ec5e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:25:24.382635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:24.381756 6389 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" podUID="a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 03 14:25:24.859629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:24.855733 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" podUID="a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 14:25:28.374975 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:25:28.374905 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:25:33.744741 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:33.744702 6389 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:25:33.747110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:33.744768 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:25:34.260327 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:34.260296 6389 generic.go:356] "Generic (PLEG): container finished" podID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerID="88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563" exitCode=0 Nov 03 14:25:34.260509 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:34.260367 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" event={"ID":"a1e41d26-58b4-42ce-a010-e2fa486614d1","Type":"ContainerDied","Data":"88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563"} Nov 03 14:25:34.260839 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:34.260822 6389 scope.go:122] "RemoveContainer" containerID="88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563" Nov 03 14:25:34.262196 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:34.262169 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:25:34.262281 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:34.262236 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:25:34.262281 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:34.262266 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:25:35.274334 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:35.274300 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" event={"ID":"a1e41d26-58b4-42ce-a010-e2fa486614d1","Type":"ContainerStarted","Data":"b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0"} Nov 03 14:25:36.063418 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:36.063383 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:25:36.063743 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:36.063720 6389 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 14:25:36.063801 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:36.063769 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 14:25:37.064196 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:37.064159 6389 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 14:25:37.064586 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:37.064222 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 14:25:38.063841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:38.063808 6389 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 14:25:38.064042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:38.063884 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 14:25:39.063952 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:39.063910 6389 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 14:25:39.064393 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:39.063985 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 14:25:40.063340 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:40.063301 6389 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 14:25:40.063588 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:40.063364 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 14:25:41.068087 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:41.068055 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:25:41.068464 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:41.068341 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:25:41.069014 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:41.068998 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:25:54.759656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.759547 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j"] Nov 03 14:25:54.763416 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.763394 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:54.779106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.779081 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-b4361f-serving-cert\"" Nov 03 14:25:54.800232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.800200 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j"] Nov 03 14:25:54.848817 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.848784 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2c6j\" (UniqueName: \"kubernetes.io/projected/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-kube-api-access-x2c6j\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:54.848981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.848833 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:54.848981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.848947 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-metrics-certs\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:54.849070 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.849056 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:54.950295 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.950255 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2c6j\" (UniqueName: \"kubernetes.io/projected/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-kube-api-access-x2c6j\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:54.950295 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.950304 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:54.950563 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.950346 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-metrics-certs\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:54.950563 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.950443 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:54.950737 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:25:54.950710 6389 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-serving-cert: secret "maas-api-e2e-extmod-b-b4361f-serving-cert" not found Nov 03 14:25:54.953863 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:25:54.950798 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-maas-api-tls podName:b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70 nodeName:}" failed. No retries permitted until 2026-11-03 14:25:55.450774753 +0000 UTC m=+831.009411232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-maas-api-tls") pod "maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" (UID: "b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70") : secret "maas-api-e2e-extmod-b-b4361f-serving-cert" not found Nov 03 14:25:54.953863 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.951821 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:54.968982 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.968944 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-metrics-certs\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:54.987465 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:54.987415 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2c6j\" (UniqueName: \"kubernetes.io/projected/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-kube-api-access-x2c6j\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:55.456932 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:55.456898 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:55.459516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:55.459492 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:55.707305 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:55.707207 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:56.034195 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:56.034167 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j"] Nov 03 14:25:56.035863 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:25:56.035838 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87a0036_e3fd_4aa5_9c1d_c4cf4a599c70.slice/crio-89db4e756cfcc9f9fa28f3d9571ae2d885cdc25c8b8d02d56b0ca096559268fa WatchSource:0}: Error finding container 89db4e756cfcc9f9fa28f3d9571ae2d885cdc25c8b8d02d56b0ca096559268fa: Status 404 returned error can't find the container with id 89db4e756cfcc9f9fa28f3d9571ae2d885cdc25c8b8d02d56b0ca096559268fa Nov 03 14:25:56.037496 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:56.037478 6389 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:25:56.606442 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:56.606406 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" event={"ID":"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70","Type":"ContainerStarted","Data":"a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41"} Nov 03 14:25:56.606600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:56.606463 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" event={"ID":"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70","Type":"ContainerStarted","Data":"89db4e756cfcc9f9fa28f3d9571ae2d885cdc25c8b8d02d56b0ca096559268fa"} Nov 03 14:25:56.606600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:56.606489 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:25:56.651319 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:25:56.651259 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" podStartSLOduration=2.259866155 podStartE2EDuration="2.651239919s" totalImagesPullingTime="391.373764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:56.647960427 +0000 UTC m=+832.206596911" watchObservedRunningTime="2026-11-03 14:25:56.651239919 +0000 UTC m=+832.209876404" Nov 03 14:25:58.376189 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:25:58.376149 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:26:00.729447 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.729416 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6"] Nov 03 14:26:00.733866 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.733845 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.749953 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.749924 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5b97f1-openshift-default-dockercfg-gbzmt\"" Nov 03 14:26:00.762917 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.762890 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6"] Nov 03 14:26:00.829633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.829295 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-istio-token\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.829633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.829381 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65281ce4-925e-4e47-82a7-850296c07953-istiod-ca-cert\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.829633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.829474 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-certs\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.829633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.829515 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-credential-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.829633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.829560 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-envoy\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.829633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.829585 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-data\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.829633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.829629 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.830175 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.829692 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65281ce4-925e-4e47-82a7-850296c07953-istio-podinfo\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.830175 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.829752 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5dch\" (UniqueName: \"kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-kube-api-access-h5dch\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.937806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.936744 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-envoy\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.937806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.936790 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-data\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.937806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.936818 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.937806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.936884 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65281ce4-925e-4e47-82a7-850296c07953-istio-podinfo\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.937806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.936951 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5dch\" (UniqueName: \"kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-kube-api-access-h5dch\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.937806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.937074 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-istio-token\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.937806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.937134 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65281ce4-925e-4e47-82a7-850296c07953-istiod-ca-cert\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.937806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.937230 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-certs\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.937806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.937272 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-credential-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.937806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.937771 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-credential-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.939328 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.938698 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-data\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.939328 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.938940 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.943580 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.939925 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65281ce4-925e-4e47-82a7-850296c07953-istiod-ca-cert\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.943580 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.940420 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-certs\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.960774 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.958394 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-envoy\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.963668 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.961646 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65281ce4-925e-4e47-82a7-850296c07953-istio-podinfo\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.968113 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.968080 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-istio-token\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:00.973342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:00.973313 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5dch\" (UniqueName: \"kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-kube-api-access-h5dch\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:01.049245 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:01.049159 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:01.384699 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:01.384577 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6"] Nov 03 14:26:01.388266 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:26:01.388234 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65281ce4_925e_4e47_82a7_850296c07953.slice/crio-59ec2d512c5c57ed066c22b7493d168a5431e180a55125457c3ad4fa59f9e975 WatchSource:0}: Error finding container 59ec2d512c5c57ed066c22b7493d168a5431e180a55125457c3ad4fa59f9e975: Status 404 returned error can't find the container with id 59ec2d512c5c57ed066c22b7493d168a5431e180a55125457c3ad4fa59f9e975 Nov 03 14:26:01.390928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:01.390887 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:26:01.391014 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:01.390961 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:26:01.391014 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:01.390996 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:26:01.662540 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:01.659156 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" event={"ID":"65281ce4-925e-4e47-82a7-850296c07953","Type":"ContainerStarted","Data":"a94ea93eea13e0582f29dd221d3d11b62fcd5d629df81adeef28d1de93ef38c5"} Nov 03 14:26:01.662540 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:01.659260 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" event={"ID":"65281ce4-925e-4e47-82a7-850296c07953","Type":"ContainerStarted","Data":"59ec2d512c5c57ed066c22b7493d168a5431e180a55125457c3ad4fa59f9e975"} Nov 03 14:26:01.746000 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:01.745938 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" podStartSLOduration=1.745915045 podStartE2EDuration="1.745915045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:26:01.740003245 +0000 UTC m=+837.298639754" watchObservedRunningTime="2026-11-03 14:26:01.745915045 +0000 UTC m=+837.304551530" Nov 03 14:26:02.049460 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:02.049420 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:02.051232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:02.051193 6389 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 14:26:02.051377 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:02.051253 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" podUID="65281ce4-925e-4e47-82a7-850296c07953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 14:26:02.694633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:02.692924 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:26:03.059301 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:03.058733 6389 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 14:26:03.059301 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:03.058797 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" podUID="65281ce4-925e-4e47-82a7-850296c07953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 14:26:04.068640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:04.061900 6389 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 14:26:04.068640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:04.061963 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" podUID="65281ce4-925e-4e47-82a7-850296c07953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 14:26:05.049452 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:05.049423 6389 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 14:26:05.049674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:05.049471 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" podUID="65281ce4-925e-4e47-82a7-850296c07953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 14:26:06.050037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:06.049990 6389 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 14:26:06.050528 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:06.050056 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" podUID="65281ce4-925e-4e47-82a7-850296c07953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 14:26:07.050320 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:07.050284 6389 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 14:26:07.050729 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:07.050348 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" podUID="65281ce4-925e-4e47-82a7-850296c07953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 14:26:08.050368 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:08.050322 6389 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 14:26:08.050944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:08.050390 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" podUID="65281ce4-925e-4e47-82a7-850296c07953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 14:26:09.054552 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:09.054519 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:09.054992 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:09.054804 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:09.055542 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:09.055523 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:26:28.374851 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:26:28.374781 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:26:52.202582 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:26:52.202546 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb"] Nov 03 14:26:58.376940 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:26:58.376901 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:27:05.068032 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:05.067997 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh"] Nov 03 14:27:07.224249 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.224183 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" podUID="a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" containerName="main" containerID="cri-o://1296d6f0cbc41e432a2486ce51a5a123221958070e82265a8bb6b5981d785a38" gracePeriod=45 Nov 03 14:27:07.444149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.444112 6389 generic.go:356] "Generic (PLEG): container finished" podID="a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" containerID="1296d6f0cbc41e432a2486ce51a5a123221958070e82265a8bb6b5981d785a38" exitCode=0 Nov 03 14:27:07.444319 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.444183 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" event={"ID":"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8","Type":"ContainerDied","Data":"1296d6f0cbc41e432a2486ce51a5a123221958070e82265a8bb6b5981d785a38"} Nov 03 14:27:07.488213 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.488149 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:27:07.610106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.610072 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-home\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-home\") pod \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " Nov 03 14:27:07.610292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.610115 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-dshm\") pod \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " Nov 03 14:27:07.610292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.610185 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-kube-api-access-8f65h\" (UniqueName: \"kubernetes.io/projected/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-kube-api-access-8f65h\") pod \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " Nov 03 14:27:07.610390 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.610359 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tls-certs\") pod \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " Nov 03 14:27:07.610390 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.610354 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-home" pod "a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" (UID: "a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:07.610490 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.610396 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-model-cache\") pod \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " Nov 03 14:27:07.610490 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.610428 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tmp-dir\") pod \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\" (UID: \"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8\") " Nov 03 14:27:07.610722 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.610678 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-model-cache" pod "a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" (UID: "a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:07.610821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.610746 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tmp-dir" pod "a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" (UID: "a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:07.610970 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.610952 6389 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-model-cache\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:07.611033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.610999 6389 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tmp-dir\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:07.611033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.611014 6389 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-home\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:07.612749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.612695 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-kube-api-access-8f65h" pod "a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" (UID: "a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8"). InnerVolumeSpecName "kube-api-access-8f65h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:07.613098 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.613077 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-dshm" pod "a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" (UID: "a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:07.613192 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.613140 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tls-certs" pod "a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" (UID: "a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:27:07.712448 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.712411 6389 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-tls-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:07.712448 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.712443 6389 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-dshm\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:07.712448 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:07.712453 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f65h\" (UniqueName: \"kubernetes.io/projected/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8-kube-api-access-8f65h\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:08.451090 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:08.451056 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" event={"ID":"a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8","Type":"ContainerDied","Data":"07f9deed731f1e0dcc57016fd80e48d18d48a9a80e939a0116af8ad0727c02ba"} Nov 03 14:27:08.451558 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:08.451100 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb" Nov 03 14:27:08.451558 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:08.451105 6389 scope.go:122] "RemoveContainer" containerID="1296d6f0cbc41e432a2486ce51a5a123221958070e82265a8bb6b5981d785a38" Nov 03 14:27:08.506914 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:08.506884 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb"] Nov 03 14:27:08.518631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:08.518575 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5b97f1/test-model-5b97f1-kserve-84558985dc-c6qfb"] Nov 03 14:27:08.619660 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:08.619573 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" path="/var/lib/kubelet/pods/a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8/volumes" Nov 03 14:27:17.448993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.448961 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858"] Nov 03 14:27:17.449565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.449551 6389 cpu_manager.go:414] "containerMap: removing container" podUID="a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" containerName="main" Nov 03 14:27:17.449627 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.449567 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" containerName="main" Nov 03 14:27:17.449704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.449694 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8cd3ee4-c080-4cd5-a6bb-4524d0ac91d8" containerName="main" Nov 03 14:27:17.453030 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.453014 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.471600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.471571 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-c88df00e610c\"" Nov 03 14:27:17.479856 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.479826 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-95ac97e0ca1e\"" Nov 03 14:27:17.479856 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.479846 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-e2e--29c79c46-dockercfg-xvjqk\"" Nov 03 14:27:17.480143 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.479859 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"openshift-service-ca.crt\"" Nov 03 14:27:17.480143 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.479879 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"kube-root-ca.crt\"" Nov 03 14:27:17.480143 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.479888 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b4361f\"" Nov 03 14:27:17.480143 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.479919 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"routing-overlay\"" Nov 03 14:27:17.493063 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.493030 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858"] Nov 03 14:27:17.531577 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.531534 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.531777 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.531738 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.531777 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.531764 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xml2r\" (UniqueName: \"kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-kube-api-access-xml2r\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.531998 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.531967 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.633327 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.633296 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.633548 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.633383 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.633548 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.633526 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.633710 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.633554 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xml2r\" (UniqueName: \"kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-kube-api-access-xml2r\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.634255 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.634229 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.634376 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.634311 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.636051 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.636033 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.659458 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.659427 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xml2r\" (UniqueName: \"kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-kube-api-access-xml2r\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:17.767019 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:17.766928 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:18.083914 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:18.083889 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858"] Nov 03 14:27:18.086239 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:27:18.086205 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f3dd10_e08b_450b_b2f2_531d97c28d77.slice/crio-d4083ece55ac4eb7c5f112ea0662fb61baeb73a181af3c0ad284ceae787097c6 WatchSource:0}: Error finding container d4083ece55ac4eb7c5f112ea0662fb61baeb73a181af3c0ad284ceae787097c6: Status 404 returned error can't find the container with id d4083ece55ac4eb7c5f112ea0662fb61baeb73a181af3c0ad284ceae787097c6 Nov 03 14:27:18.605956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:18.605917 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" event={"ID":"92f3dd10-e08b-450b-b2f2-531d97c28d77","Type":"ContainerStarted","Data":"1bcd9e6d38979887b93a4ada878ec219a20b74400f9d6c464963b99e0deea206"} Nov 03 14:27:18.606486 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:18.605984 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" event={"ID":"92f3dd10-e08b-450b-b2f2-531d97c28d77","Type":"ContainerStarted","Data":"d4083ece55ac4eb7c5f112ea0662fb61baeb73a181af3c0ad284ceae787097c6"} Nov 03 14:27:18.606486 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:18.606012 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:18.649037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:18.648990 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" podStartSLOduration=1.6489752389999999 podStartE2EDuration="1.648975239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:18.647771987 +0000 UTC m=+914.206408484" watchObservedRunningTime="2026-11-03 14:27:18.648975239 +0000 UTC m=+914.207611723" Nov 03 14:27:18.869471 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:18.869373 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t"] Nov 03 14:27:18.911387 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:18.911353 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59"] Nov 03 14:27:20.091426 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.091389 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" podUID="20313390-9803-4c52-898e-23ce0fed0365" containerName="main" containerID="cri-o://379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524" gracePeriod=45 Nov 03 14:27:20.368310 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.368280 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:27:20.474199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.474166 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-dshm\") pod \"20313390-9803-4c52-898e-23ce0fed0365\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " Nov 03 14:27:20.474199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.474199 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-tmp-dir\") pod \"20313390-9803-4c52-898e-23ce0fed0365\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " Nov 03 14:27:20.474451 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.474244 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20313390-9803-4c52-898e-23ce0fed0365-kube-api-access-dmnql\" (UniqueName: \"kubernetes.io/projected/20313390-9803-4c52-898e-23ce0fed0365-kube-api-access-dmnql\") pod \"20313390-9803-4c52-898e-23ce0fed0365\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " Nov 03 14:27:20.474451 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.474290 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-home\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-home\") pod \"20313390-9803-4c52-898e-23ce0fed0365\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " Nov 03 14:27:20.474451 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.474354 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20313390-9803-4c52-898e-23ce0fed0365-tls-certs\" (UniqueName: \"kubernetes.io/secret/20313390-9803-4c52-898e-23ce0fed0365-tls-certs\") pod \"20313390-9803-4c52-898e-23ce0fed0365\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " Nov 03 14:27:20.474451 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.474427 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-model-cache\") pod \"20313390-9803-4c52-898e-23ce0fed0365\" (UID: \"20313390-9803-4c52-898e-23ce0fed0365\") " Nov 03 14:27:20.474687 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.474508 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-tmp-dir" pod "20313390-9803-4c52-898e-23ce0fed0365" (UID: "20313390-9803-4c52-898e-23ce0fed0365"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:20.474687 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.474655 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-home" pod "20313390-9803-4c52-898e-23ce0fed0365" (UID: "20313390-9803-4c52-898e-23ce0fed0365"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:20.474794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.474766 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-model-cache" pod "20313390-9803-4c52-898e-23ce0fed0365" (UID: "20313390-9803-4c52-898e-23ce0fed0365"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:20.475028 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.475011 6389 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-model-cache\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:20.475080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.475037 6389 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-tmp-dir\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:20.475080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.475055 6389 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-home\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:20.477186 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.477159 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20313390-9803-4c52-898e-23ce0fed0365-kube-api-access-dmnql" pod "20313390-9803-4c52-898e-23ce0fed0365" (UID: "20313390-9803-4c52-898e-23ce0fed0365"). InnerVolumeSpecName "kube-api-access-dmnql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:20.477294 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.477194 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20313390-9803-4c52-898e-23ce0fed0365-tls-certs" pod "20313390-9803-4c52-898e-23ce0fed0365" (UID: "20313390-9803-4c52-898e-23ce0fed0365"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:27:20.477294 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.477272 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-dshm" pod "20313390-9803-4c52-898e-23ce0fed0365" (UID: "20313390-9803-4c52-898e-23ce0fed0365"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:20.575808 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.575775 6389 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20313390-9803-4c52-898e-23ce0fed0365-dshm\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:20.575808 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.575803 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmnql\" (UniqueName: \"kubernetes.io/projected/20313390-9803-4c52-898e-23ce0fed0365-kube-api-access-dmnql\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:20.575808 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.575815 6389 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20313390-9803-4c52-898e-23ce0fed0365-tls-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:20.620388 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.620292 6389 generic.go:356] "Generic (PLEG): container finished" podID="20313390-9803-4c52-898e-23ce0fed0365" containerID="379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524" exitCode=0 Nov 03 14:27:20.620388 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.620351 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" Nov 03 14:27:20.620635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.620389 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" event={"ID":"20313390-9803-4c52-898e-23ce0fed0365","Type":"ContainerDied","Data":"379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524"} Nov 03 14:27:20.620635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.620449 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh" event={"ID":"20313390-9803-4c52-898e-23ce0fed0365","Type":"ContainerDied","Data":"14e59a6d451a7f8c5de99e26707cc57ae31d59627444c8bd38e2c6611a430b62"} Nov 03 14:27:20.620635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.620471 6389 scope.go:122] "RemoveContainer" containerID="379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524" Nov 03 14:27:20.635155 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.635134 6389 scope.go:122] "RemoveContainer" containerID="379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524" Nov 03 14:27:20.635735 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:27:20.635705 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524\": container with ID starting with 379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524 not found: ID does not exist" containerID="379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524" Nov 03 14:27:20.635854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.635764 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524"} err="failed to get container status \"379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524\": rpc error: code = NotFound desc = could not find container \"379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524\": container with ID starting with 379c1c983ab3c6f073be60d98ee8e7c3c6acf88d524743edd90e2561cc5d8524 not found: ID does not exist" Nov 03 14:27:20.676974 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.676941 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh"] Nov 03 14:27:20.681671 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:20.681647 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e759e5/test-model-e759e5-kserve-584d466575-r5grh"] Nov 03 14:27:22.631635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:22.631511 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20313390-9803-4c52-898e-23ce0fed0365" path="/var/lib/kubelet/pods/20313390-9803-4c52-898e-23ce0fed0365/volumes" Nov 03 14:27:23.881638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:23.869679 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" podUID="2d7ec965-e58c-423a-9772-5d3cfa5063a5" containerName="payload-processing" containerID="cri-o://0278a161be0a79daf5f8f1978a4d504532979028d55b0b1cf6be1f1febbe8980" gracePeriod=25 Nov 03 14:27:23.915662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:23.914686 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" podUID="2142f874-bf66-4f25-8b7b-f26204387a6c" containerName="payload-pre-processing" containerID="cri-o://54f5934287357f058dcfe10839fcb1a97a3045c02e1e4b6fff9d8b0aedf73d87" gracePeriod=25 Nov 03 14:27:25.985497 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:25.985466 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482"] Nov 03 14:27:25.986144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:25.986128 6389 cpu_manager.go:414] "containerMap: removing container" podUID="20313390-9803-4c52-898e-23ce0fed0365" containerName="main" Nov 03 14:27:25.986196 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:25.986145 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20313390-9803-4c52-898e-23ce0fed0365" containerName="main" Nov 03 14:27:25.986255 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:25.986246 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="20313390-9803-4c52-898e-23ce0fed0365" containerName="main" Nov 03 14:27:25.990998 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:25.990979 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.019902 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.019875 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e759e5-openshift-default-dockercfg-ldb9g\"" Nov 03 14:27:26.023788 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.023749 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482"] Nov 03 14:27:26.041258 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.041224 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-data\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.041453 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.041272 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-podinfo\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.041453 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.041349 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-certs\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.041453 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.041392 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-credential-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.041564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.041465 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blfd7\" (UniqueName: \"kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-kube-api-access-blfd7\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.041564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.041527 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-token\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.041564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.041555 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istiod-ca-cert\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.041810 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.041671 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.041810 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.041708 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-envoy\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.142997 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.142963 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-data\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.143175 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.143005 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-podinfo\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.143175 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.143084 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-certs\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.143175 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.143128 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-credential-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.143370 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.143210 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blfd7\" (UniqueName: \"kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-kube-api-access-blfd7\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.143370 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.143290 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-token\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.143370 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.143321 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istiod-ca-cert\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.143562 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.143402 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.143562 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.143429 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-data\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.143562 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.143438 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-envoy\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.143562 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.143538 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-certs\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.143820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.143579 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-credential-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.144080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.144055 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.144718 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.144697 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istiod-ca-cert\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.145757 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.145737 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-envoy\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.159176 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.159152 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-podinfo\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.170843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.170806 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blfd7\" (UniqueName: \"kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-kube-api-access-blfd7\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.171122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.171103 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-token\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.305382 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.305296 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:26.638564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.638542 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482"] Nov 03 14:27:26.640301 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:27:26.640275 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1bf405_ee22_4a6d_9ff6_f0651334075e.slice/crio-1cf02a989239b2c2b7a5e35032da70e26763d791adac6ad68d69abc0bc90c043 WatchSource:0}: Error finding container 1cf02a989239b2c2b7a5e35032da70e26763d791adac6ad68d69abc0bc90c043: Status 404 returned error can't find the container with id 1cf02a989239b2c2b7a5e35032da70e26763d791adac6ad68d69abc0bc90c043 Nov 03 14:27:26.642796 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.642766 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:27:26.642877 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.642831 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:27:26.642877 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.642859 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:27:26.828169 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.828118 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" event={"ID":"6b1bf405-ee22-4a6d-9ff6-f0651334075e","Type":"ContainerStarted","Data":"7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414"} Nov 03 14:27:26.828438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.828208 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" event={"ID":"6b1bf405-ee22-4a6d-9ff6-f0651334075e","Type":"ContainerStarted","Data":"1cf02a989239b2c2b7a5e35032da70e26763d791adac6ad68d69abc0bc90c043"} Nov 03 14:27:26.879354 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:26.879306 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" podStartSLOduration=1.879291837 podStartE2EDuration="1.879291837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:26.87800028 +0000 UTC m=+922.436636764" watchObservedRunningTime="2026-11-03 14:27:26.879291837 +0000 UTC m=+922.437928322" Nov 03 14:27:27.305881 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:27.305846 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:27.307657 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:27.307631 6389 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 14:27:27.307779 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:27.307689 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" podUID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 14:27:28.306131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:28.306094 6389 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 14:27:28.306525 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:28.306162 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" podUID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 14:27:28.373773 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:27:28.373736 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:27:29.306695 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:29.306651 6389 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 14:27:29.307208 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:29.306747 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" podUID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 14:27:29.614965 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:29.614885 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:27:30.305780 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:30.305749 6389 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 14:27:30.305942 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:30.305809 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" podUID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 14:27:30.699215 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:30.699171 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx"] Nov 03 14:27:30.699640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:30.699423 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" podUID="dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b" containerName="maas-api" containerID="cri-o://c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93" gracePeriod=30 Nov 03 14:27:31.367708 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.360429 6389 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 14:27:31.367708 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.360490 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" podUID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 14:27:31.455768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.455396 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:27:31.465073 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.465036 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-openshift-service-ca\") pod \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " Nov 03 14:27:31.465241 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.465097 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-kube-api-access-x2gzz\" (UniqueName: \"kubernetes.io/projected/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-kube-api-access-x2gzz\") pod \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " Nov 03 14:27:31.465295 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.465269 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-metrics-certs\") pod \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " Nov 03 14:27:31.465360 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.465334 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-maas-api-tls\") pod \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\" (UID: \"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b\") " Nov 03 14:27:31.465498 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.465468 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-openshift-service-ca" pod "dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b" (UID: "dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:31.465861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.465838 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:31.467652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.467629 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-maas-api-tls" pod "dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b" (UID: "dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:27:31.467652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.467630 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-kube-api-access-x2gzz" pod "dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b" (UID: "dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b"). InnerVolumeSpecName "kube-api-access-x2gzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:31.467786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.467664 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-metrics-certs" pod "dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b" (UID: "dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:27:31.567304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.567271 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gzz\" (UniqueName: \"kubernetes.io/projected/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-kube-api-access-x2gzz\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:31.567304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.567298 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:31.567304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.567308 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:31.908325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.908288 6389 generic.go:356] "Generic (PLEG): container finished" podID="dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b" containerID="c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93" exitCode=0 Nov 03 14:27:31.908819 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.908382 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" event={"ID":"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b","Type":"ContainerDied","Data":"c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93"} Nov 03 14:27:31.908819 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.908434 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" event={"ID":"dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b","Type":"ContainerDied","Data":"c1bd5389b055b7de9a2d976eda7be7abdc279126ebbb039f42eba9ce5b0dfdb2"} Nov 03 14:27:31.908819 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.908466 6389 scope.go:122] "RemoveContainer" containerID="c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93" Nov 03 14:27:31.908819 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.908683 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx" Nov 03 14:27:31.957186 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.957079 6389 scope.go:122] "RemoveContainer" containerID="c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93" Nov 03 14:27:31.957503 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:27:31.957481 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93\": container with ID starting with c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93 not found: ID does not exist" containerID="c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93" Nov 03 14:27:31.957597 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.957514 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93"} err="failed to get container status \"c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93\": rpc error: code = NotFound desc = could not find container \"c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93\": container with ID starting with c4d649bcaecf46a57e8d53f6d38e9cd9bc4fe93710984eac4957d526a577ed93 not found: ID does not exist" Nov 03 14:27:31.982813 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.982783 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx"] Nov 03 14:27:31.990523 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:31.990499 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5b97f1-84bf489bdb-f9pcx"] Nov 03 14:27:32.310633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:32.306676 6389 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 14:27:32.310633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:32.306745 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" podUID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 14:27:32.618387 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:32.618311 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b" path="/var/lib/kubelet/pods/dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b/volumes" Nov 03 14:27:32.941815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:32.941773 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" podUID="2d7ec965-e58c-423a-9772-5d3cfa5063a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:27:32.942363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:32.941868 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" podUID="2142f874-bf66-4f25-8b7b-f26204387a6c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:27:33.310454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:33.310372 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:33.310809 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:33.310787 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:33.311402 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:33.311385 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:27:34.890073 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:34.890040 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7"] Nov 03 14:27:34.890592 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:34.890244 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerName="istio-proxy" containerID="cri-o://9d451ec677fba2760dfaca2e36afbd477810c18918139944c51748ec98ff41da" gracePeriod=30 Nov 03 14:27:40.015928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.015899 6389 generic.go:356] "Generic (PLEG): container finished" podID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerID="9d451ec677fba2760dfaca2e36afbd477810c18918139944c51748ec98ff41da" exitCode=0 Nov 03 14:27:40.016390 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.015957 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" event={"ID":"2e4314dc-1418-4cf6-ba99-4e3f448e3cab","Type":"ContainerDied","Data":"9d451ec677fba2760dfaca2e36afbd477810c18918139944c51748ec98ff41da"} Nov 03 14:27:40.161720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.161661 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:27:40.249382 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.249351 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-credential-socket\") pod \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " Nov 03 14:27:40.249581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.249403 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-podinfo\") pod \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " Nov 03 14:27:40.249581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.249459 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istiod-ca-cert\") pod \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " Nov 03 14:27:40.249581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.249541 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-envoy\") pod \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " Nov 03 14:27:40.249744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.249585 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-certs\") pod \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " Nov 03 14:27:40.249744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.249639 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-data\") pod \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " Nov 03 14:27:40.249744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.249679 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-token\" (UniqueName: \"kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-token\") pod \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " Nov 03 14:27:40.249744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.249687 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-credential-socket" pod "2e4314dc-1418-4cf6-ba99-4e3f448e3cab" (UID: "2e4314dc-1418-4cf6-ba99-4e3f448e3cab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:40.249744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.249716 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-socket\") pod \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " Nov 03 14:27:40.249934 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.249885 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-certs" pod "2e4314dc-1418-4cf6-ba99-4e3f448e3cab" (UID: "2e4314dc-1418-4cf6-ba99-4e3f448e3cab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:40.249934 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.249887 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-kube-api-access-h4465\" (UniqueName: \"kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-kube-api-access-h4465\") pod \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\" (UID: \"2e4314dc-1418-4cf6-ba99-4e3f448e3cab\") " Nov 03 14:27:40.249934 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.249920 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istiod-ca-cert" pod "2e4314dc-1418-4cf6-ba99-4e3f448e3cab" (UID: "2e4314dc-1418-4cf6-ba99-4e3f448e3cab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:40.250191 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.250169 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-socket" pod "2e4314dc-1418-4cf6-ba99-4e3f448e3cab" (UID: "2e4314dc-1418-4cf6-ba99-4e3f448e3cab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:40.251122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.251001 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:40.251122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.251033 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:40.251122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.251062 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:40.251122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.251077 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:40.261857 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.254448 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-data" pod "2e4314dc-1418-4cf6-ba99-4e3f448e3cab" (UID: "2e4314dc-1418-4cf6-ba99-4e3f448e3cab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:40.264010 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.263967 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-kube-api-access-h4465" pod "2e4314dc-1418-4cf6-ba99-4e3f448e3cab" (UID: "2e4314dc-1418-4cf6-ba99-4e3f448e3cab"). InnerVolumeSpecName "kube-api-access-h4465". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:40.264320 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.264284 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-token" pod "2e4314dc-1418-4cf6-ba99-4e3f448e3cab" (UID: "2e4314dc-1418-4cf6-ba99-4e3f448e3cab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:40.264586 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.264559 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-podinfo" pod "2e4314dc-1418-4cf6-ba99-4e3f448e3cab" (UID: "2e4314dc-1418-4cf6-ba99-4e3f448e3cab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:27:40.266223 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.266198 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-envoy" pod "2e4314dc-1418-4cf6-ba99-4e3f448e3cab" (UID: "2e4314dc-1418-4cf6-ba99-4e3f448e3cab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:40.352242 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.352209 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4465\" (UniqueName: \"kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-kube-api-access-h4465\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:40.352242 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.352241 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:40.352242 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.352251 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:40.352473 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.352260 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:40.352473 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:40.352267 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4314dc-1418-4cf6-ba99-4e3f448e3cab-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:41.024350 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:41.024320 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" Nov 03 14:27:41.024834 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:41.024320 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7" event={"ID":"2e4314dc-1418-4cf6-ba99-4e3f448e3cab","Type":"ContainerDied","Data":"6b197985312316143855e5e415a0ace4b975068c70b599b6a85bb02ccb8a15be"} Nov 03 14:27:41.024834 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:41.024459 6389 scope.go:122] "RemoveContainer" containerID="9d451ec677fba2760dfaca2e36afbd477810c18918139944c51748ec98ff41da" Nov 03 14:27:41.115532 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:41.115453 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7"] Nov 03 14:27:41.118008 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:41.117978 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-92rg7"] Nov 03 14:27:42.620765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:42.620732 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" path="/var/lib/kubelet/pods/2e4314dc-1418-4cf6-ba99-4e3f448e3cab/volumes" Nov 03 14:27:42.941556 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:42.941515 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" podUID="2d7ec965-e58c-423a-9772-5d3cfa5063a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:27:42.941791 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:42.941516 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" podUID="2142f874-bf66-4f25-8b7b-f26204387a6c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:27:44.047131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.047092 6389 generic.go:356] "Generic (PLEG): container finished" podID="2142f874-bf66-4f25-8b7b-f26204387a6c" containerID="54f5934287357f058dcfe10839fcb1a97a3045c02e1e4b6fff9d8b0aedf73d87" exitCode=0 Nov 03 14:27:44.047659 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.047264 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" event={"ID":"2142f874-bf66-4f25-8b7b-f26204387a6c","Type":"ContainerDied","Data":"54f5934287357f058dcfe10839fcb1a97a3045c02e1e4b6fff9d8b0aedf73d87"} Nov 03 14:27:44.049949 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.049920 6389 generic.go:356] "Generic (PLEG): container finished" podID="2d7ec965-e58c-423a-9772-5d3cfa5063a5" containerID="0278a161be0a79daf5f8f1978a4d504532979028d55b0b1cf6be1f1febbe8980" exitCode=0 Nov 03 14:27:44.050102 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.050084 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" event={"ID":"2d7ec965-e58c-423a-9772-5d3cfa5063a5","Type":"ContainerDied","Data":"0278a161be0a79daf5f8f1978a4d504532979028d55b0b1cf6be1f1febbe8980"} Nov 03 14:27:44.073718 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.073685 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" Nov 03 14:27:44.138987 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.138923 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" Nov 03 14:27:44.217499 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.217469 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2142f874-bf66-4f25-8b7b-f26204387a6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2142f874-bf66-4f25-8b7b-f26204387a6c-plugins-config-volume\") pod \"2142f874-bf66-4f25-8b7b-f26204387a6c\" (UID: \"2142f874-bf66-4f25-8b7b-f26204387a6c\") " Nov 03 14:27:44.217689 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.217538 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7ec965-e58c-423a-9772-5d3cfa5063a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7ec965-e58c-423a-9772-5d3cfa5063a5-plugins-config-volume\") pod \"2d7ec965-e58c-423a-9772-5d3cfa5063a5\" (UID: \"2d7ec965-e58c-423a-9772-5d3cfa5063a5\") " Nov 03 14:27:44.217746 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.217694 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2142f874-bf66-4f25-8b7b-f26204387a6c-kube-api-access-tcb7p\" (UniqueName: \"kubernetes.io/projected/2142f874-bf66-4f25-8b7b-f26204387a6c-kube-api-access-tcb7p\") pod \"2142f874-bf66-4f25-8b7b-f26204387a6c\" (UID: \"2142f874-bf66-4f25-8b7b-f26204387a6c\") " Nov 03 14:27:44.217893 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.217874 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7ec965-e58c-423a-9772-5d3cfa5063a5-kube-api-access-tbzq4\" (UniqueName: \"kubernetes.io/projected/2d7ec965-e58c-423a-9772-5d3cfa5063a5-kube-api-access-tbzq4\") pod \"2d7ec965-e58c-423a-9772-5d3cfa5063a5\" (UID: \"2d7ec965-e58c-423a-9772-5d3cfa5063a5\") " Nov 03 14:27:44.218120 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.217984 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2142f874-bf66-4f25-8b7b-f26204387a6c-plugins-config-volume" pod "2142f874-bf66-4f25-8b7b-f26204387a6c" (UID: "2142f874-bf66-4f25-8b7b-f26204387a6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:44.218120 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.218052 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7ec965-e58c-423a-9772-5d3cfa5063a5-plugins-config-volume" pod "2d7ec965-e58c-423a-9772-5d3cfa5063a5" (UID: "2d7ec965-e58c-423a-9772-5d3cfa5063a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:44.218378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.218354 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2142f874-bf66-4f25-8b7b-f26204387a6c-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:44.218451 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.218385 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7ec965-e58c-423a-9772-5d3cfa5063a5-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:44.220190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.220168 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7ec965-e58c-423a-9772-5d3cfa5063a5-kube-api-access-tbzq4" pod "2d7ec965-e58c-423a-9772-5d3cfa5063a5" (UID: "2d7ec965-e58c-423a-9772-5d3cfa5063a5"). InnerVolumeSpecName "kube-api-access-tbzq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:44.220299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.220186 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2142f874-bf66-4f25-8b7b-f26204387a6c-kube-api-access-tcb7p" pod "2142f874-bf66-4f25-8b7b-f26204387a6c" (UID: "2142f874-bf66-4f25-8b7b-f26204387a6c"). InnerVolumeSpecName "kube-api-access-tcb7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:44.319323 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.319290 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbzq4\" (UniqueName: \"kubernetes.io/projected/2d7ec965-e58c-423a-9772-5d3cfa5063a5-kube-api-access-tbzq4\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:44.319323 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:44.319321 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcb7p\" (UniqueName: \"kubernetes.io/projected/2142f874-bf66-4f25-8b7b-f26204387a6c-kube-api-access-tcb7p\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:45.058427 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:45.058400 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" Nov 03 14:27:45.058427 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:45.058421 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59" event={"ID":"2142f874-bf66-4f25-8b7b-f26204387a6c","Type":"ContainerDied","Data":"d71a3fd9802f527b74e69744b8ddfc026ad7db5510c4bff691de61e23c8e71b8"} Nov 03 14:27:45.059036 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:45.058476 6389 scope.go:122] "RemoveContainer" containerID="54f5934287357f058dcfe10839fcb1a97a3045c02e1e4b6fff9d8b0aedf73d87" Nov 03 14:27:45.060183 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:45.060151 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" Nov 03 14:27:45.060279 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:45.060158 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t" event={"ID":"2d7ec965-e58c-423a-9772-5d3cfa5063a5","Type":"ContainerDied","Data":"78a6e831738ccb53af799c64ed93a0a5e1408136717cde17f3f50c1ddfe54c4f"} Nov 03 14:27:45.068502 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:45.068483 6389 scope.go:122] "RemoveContainer" containerID="0278a161be0a79daf5f8f1978a4d504532979028d55b0b1cf6be1f1febbe8980" Nov 03 14:27:45.204683 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:45.191300 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59"] Nov 03 14:27:45.205729 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:45.205684 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5b97f1-56995974c5-hnm59"] Nov 03 14:27:45.241501 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:45.241466 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t"] Nov 03 14:27:45.247062 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:45.247032 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5b97f1-7954c5f865-hks4t"] Nov 03 14:27:46.083665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:46.083634 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6"] Nov 03 14:27:46.084170 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:46.083856 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" podUID="65281ce4-925e-4e47-82a7-850296c07953" containerName="istio-proxy" containerID="cri-o://a94ea93eea13e0582f29dd221d3d11b62fcd5d629df81adeef28d1de93ef38c5" gracePeriod=30 Nov 03 14:27:46.618990 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:46.618958 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2142f874-bf66-4f25-8b7b-f26204387a6c" path="/var/lib/kubelet/pods/2142f874-bf66-4f25-8b7b-f26204387a6c/volumes" Nov 03 14:27:46.619506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:46.619490 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7ec965-e58c-423a-9772-5d3cfa5063a5" path="/var/lib/kubelet/pods/2d7ec965-e58c-423a-9772-5d3cfa5063a5/volumes" Nov 03 14:27:47.402640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.402599 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5"] Nov 03 14:27:47.403209 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.403184 6389 cpu_manager.go:414] "containerMap: removing container" podUID="2142f874-bf66-4f25-8b7b-f26204387a6c" containerName="payload-pre-processing" Nov 03 14:27:47.403209 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.403209 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2142f874-bf66-4f25-8b7b-f26204387a6c" containerName="payload-pre-processing" Nov 03 14:27:47.403342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.403237 6389 cpu_manager.go:414] "containerMap: removing container" podUID="2d7ec965-e58c-423a-9772-5d3cfa5063a5" containerName="payload-processing" Nov 03 14:27:47.403342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.403243 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7ec965-e58c-423a-9772-5d3cfa5063a5" containerName="payload-processing" Nov 03 14:27:47.403342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.403250 6389 cpu_manager.go:414] "containerMap: removing container" podUID="dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b" containerName="maas-api" Nov 03 14:27:47.403342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.403255 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b" containerName="maas-api" Nov 03 14:27:47.403342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.403267 6389 cpu_manager.go:414] "containerMap: removing container" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerName="istio-proxy" Nov 03 14:27:47.403342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.403271 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerName="istio-proxy" Nov 03 14:27:47.403581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.403372 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="2142f874-bf66-4f25-8b7b-f26204387a6c" containerName="payload-pre-processing" Nov 03 14:27:47.403581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.403384 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7ec965-e58c-423a-9772-5d3cfa5063a5" containerName="payload-processing" Nov 03 14:27:47.403581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.403389 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6f4ffd-b75d-4b26-a2a9-9e9ef051654b" containerName="maas-api" Nov 03 14:27:47.403581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.403396 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4314dc-1418-4cf6-ba99-4e3f448e3cab" containerName="istio-proxy" Nov 03 14:27:47.407166 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.407147 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.441674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.441640 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-5534d1ee322d\"" Nov 03 14:27:47.441868 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.441679 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-e2e--e9e6f325-dockercfg-54lvq\"" Nov 03 14:27:47.441868 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.441759 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"kube-root-ca.crt\"" Nov 03 14:27:47.441868 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.441800 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-331c9e\"" Nov 03 14:27:47.442113 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.442091 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"routing-overlay\"" Nov 03 14:27:47.442200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.442185 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"openshift-service-ca.crt\"" Nov 03 14:27:47.444585 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.444561 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5"] Nov 03 14:27:47.496026 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.495990 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvfr4\" (UniqueName: \"kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-kube-api-access-dvfr4\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.496219 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.496056 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.496219 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.496130 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.496320 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.496300 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.597398 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.597364 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.597601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.597443 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvfr4\" (UniqueName: \"kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-kube-api-access-dvfr4\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.597601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.597494 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.597601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.597527 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.598255 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.598228 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.598334 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.598317 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.600127 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.600109 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.619428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.619401 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvfr4\" (UniqueName: \"kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-kube-api-access-dvfr4\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:47.727211 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:47.726623 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:48.078001 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:48.077966 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5"] Nov 03 14:27:48.079828 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:27:48.079790 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94c5fe4_b016_471c_bc6d_410c843851f0.slice/crio-ea3297c957ae721acc2ca69148fc0a340451d7afd96688dd6a2d0f5ffc0f3a7c WatchSource:0}: Error finding container ea3297c957ae721acc2ca69148fc0a340451d7afd96688dd6a2d0f5ffc0f3a7c: Status 404 returned error can't find the container with id ea3297c957ae721acc2ca69148fc0a340451d7afd96688dd6a2d0f5ffc0f3a7c Nov 03 14:27:48.088165 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:48.088138 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" event={"ID":"e94c5fe4-b016-471c-bc6d-410c843851f0","Type":"ContainerStarted","Data":"ea3297c957ae721acc2ca69148fc0a340451d7afd96688dd6a2d0f5ffc0f3a7c"} Nov 03 14:27:49.095720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:49.095682 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" event={"ID":"e94c5fe4-b016-471c-bc6d-410c843851f0","Type":"ContainerStarted","Data":"59a0b5fb02946d621805bdf7d8166a0b76750143f4a25d3cf1b325f2507db012"} Nov 03 14:27:49.096162 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:49.095738 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:49.141142 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:49.141086 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" podStartSLOduration=2.1410670019999998 podStartE2EDuration="2.141067002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:49.140733341 +0000 UTC m=+944.699369826" watchObservedRunningTime="2026-11-03 14:27:49.141067002 +0000 UTC m=+944.699703488" Nov 03 14:27:49.916292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:49.916260 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f"] Nov 03 14:27:49.941281 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:49.941251 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.064706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:49.982664 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f"] Nov 03 14:27:50.064706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:49.983224 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-dshm\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.064706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.063669 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-home\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.064706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.063748 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-model-cache\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.064706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.063791 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jzw\" (UniqueName: \"kubernetes.io/projected/151733fd-8ae3-4401-870c-8384e18d768b-kube-api-access-68jzw\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.064706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.063909 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-tmp-dir\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.064706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.064003 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151733fd-8ae3-4401-870c-8384e18d768b-tls-certs\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.064706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:49.984687 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-86c2e2\"/\"distinct-2-w0-86c2e2-kserve-self-signed-certs\"" Nov 03 14:27:50.172641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.169106 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151733fd-8ae3-4401-870c-8384e18d768b-tls-certs\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.172641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.169226 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-dshm\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.172641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.169256 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-home\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.172641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.169315 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-model-cache\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.172641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.169348 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jzw\" (UniqueName: \"kubernetes.io/projected/151733fd-8ae3-4401-870c-8384e18d768b-kube-api-access-68jzw\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.172641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.169441 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-tmp-dir\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.172641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.169965 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-tmp-dir\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.172641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.171246 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-home\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.172641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.171826 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-model-cache\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.267633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.267007 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-dshm\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.276095 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.274265 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151733fd-8ae3-4401-870c-8384e18d768b-tls-certs\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.296632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.296574 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jzw\" (UniqueName: \"kubernetes.io/projected/151733fd-8ae3-4401-870c-8384e18d768b-kube-api-access-68jzw\") pod \"distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:50.586852 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:50.586751 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:51.924635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:51.778109 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f"] Nov 03 14:27:52.316515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.313845 6389 generic.go:356] "Generic (PLEG): container finished" podID="65281ce4-925e-4e47-82a7-850296c07953" containerID="a94ea93eea13e0582f29dd221d3d11b62fcd5d629df81adeef28d1de93ef38c5" exitCode=0 Nov 03 14:27:52.316515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.314021 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" event={"ID":"65281ce4-925e-4e47-82a7-850296c07953","Type":"ContainerDied","Data":"a94ea93eea13e0582f29dd221d3d11b62fcd5d629df81adeef28d1de93ef38c5"} Nov 03 14:27:52.340634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.331841 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" event={"ID":"151733fd-8ae3-4401-870c-8384e18d768b","Type":"ContainerStarted","Data":"312fc0fc64126f340e0a75973a355096230a391d018f09d70f13e91eb5cb9eba"} Nov 03 14:27:52.897507 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.895573 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:27:52.966631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.966570 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65281ce4-925e-4e47-82a7-850296c07953-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65281ce4-925e-4e47-82a7-850296c07953-istio-podinfo\") pod \"65281ce4-925e-4e47-82a7-850296c07953\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " Nov 03 14:27:52.967107 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.966645 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-envoy\") pod \"65281ce4-925e-4e47-82a7-850296c07953\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " Nov 03 14:27:52.967107 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.966711 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-socket\") pod \"65281ce4-925e-4e47-82a7-850296c07953\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " Nov 03 14:27:52.967107 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.966886 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-istio-token\" (UniqueName: \"kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-istio-token\") pod \"65281ce4-925e-4e47-82a7-850296c07953\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " Nov 03 14:27:52.967107 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.966916 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65281ce4-925e-4e47-82a7-850296c07953-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65281ce4-925e-4e47-82a7-850296c07953-istiod-ca-cert\") pod \"65281ce4-925e-4e47-82a7-850296c07953\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " Nov 03 14:27:52.967107 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.966961 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-certs\") pod \"65281ce4-925e-4e47-82a7-850296c07953\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " Nov 03 14:27:52.967383 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.967119 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-credential-socket\") pod \"65281ce4-925e-4e47-82a7-850296c07953\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " Nov 03 14:27:52.967383 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.967148 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-data\") pod \"65281ce4-925e-4e47-82a7-850296c07953\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " Nov 03 14:27:52.967383 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.967222 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-kube-api-access-h5dch\" (UniqueName: \"kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-kube-api-access-h5dch\") pod \"65281ce4-925e-4e47-82a7-850296c07953\" (UID: \"65281ce4-925e-4e47-82a7-850296c07953\") " Nov 03 14:27:52.969870 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.969814 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-credential-socket" pod "65281ce4-925e-4e47-82a7-850296c07953" (UID: "65281ce4-925e-4e47-82a7-850296c07953"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:52.970430 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.970388 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65281ce4-925e-4e47-82a7-850296c07953-istiod-ca-cert" pod "65281ce4-925e-4e47-82a7-850296c07953" (UID: "65281ce4-925e-4e47-82a7-850296c07953"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:52.970826 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.970776 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-data" pod "65281ce4-925e-4e47-82a7-850296c07953" (UID: "65281ce4-925e-4e47-82a7-850296c07953"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:52.971038 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.971001 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-socket" pod "65281ce4-925e-4e47-82a7-850296c07953" (UID: "65281ce4-925e-4e47-82a7-850296c07953"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:52.971342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.971322 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-certs" pod "65281ce4-925e-4e47-82a7-850296c07953" (UID: "65281ce4-925e-4e47-82a7-850296c07953"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:53.000976 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.997336 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65281ce4-925e-4e47-82a7-850296c07953-istio-podinfo" pod "65281ce4-925e-4e47-82a7-850296c07953" (UID: "65281ce4-925e-4e47-82a7-850296c07953"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:27:53.000976 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:52.999712 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-envoy" pod "65281ce4-925e-4e47-82a7-850296c07953" (UID: "65281ce4-925e-4e47-82a7-850296c07953"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:53.002749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.002673 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-istio-token" pod "65281ce4-925e-4e47-82a7-850296c07953" (UID: "65281ce4-925e-4e47-82a7-850296c07953"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:53.006677 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.006561 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-kube-api-access-h5dch" pod "65281ce4-925e-4e47-82a7-850296c07953" (UID: "65281ce4-925e-4e47-82a7-850296c07953"). InnerVolumeSpecName "kube-api-access-h5dch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:53.069138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.069039 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:53.069138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.069071 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:53.069138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.069080 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65281ce4-925e-4e47-82a7-850296c07953-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:53.069138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.069090 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:53.069138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.069098 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:53.069138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.069107 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:53.069138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.069115 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5dch\" (UniqueName: \"kubernetes.io/projected/65281ce4-925e-4e47-82a7-850296c07953-kube-api-access-h5dch\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:53.069138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.069123 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65281ce4-925e-4e47-82a7-850296c07953-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:53.069138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.069131 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65281ce4-925e-4e47-82a7-850296c07953-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:53.188783 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.188738 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5"] Nov 03 14:27:53.191937 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.191910 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:53.339678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.339552 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" event={"ID":"65281ce4-925e-4e47-82a7-850296c07953","Type":"ContainerDied","Data":"59ec2d512c5c57ed066c22b7493d168a5431e180a55125457c3ad4fa59f9e975"} Nov 03 14:27:53.339678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.339572 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6" Nov 03 14:27:53.339678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.339643 6389 scope.go:122] "RemoveContainer" containerID="a94ea93eea13e0582f29dd221d3d11b62fcd5d629df81adeef28d1de93ef38c5" Nov 03 14:27:53.341814 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.341791 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" event={"ID":"151733fd-8ae3-4401-870c-8384e18d768b","Type":"ContainerStarted","Data":"4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc"} Nov 03 14:27:53.399372 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.399318 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" podStartSLOduration=4.399304654 podStartE2EDuration="4.399304654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:53.39795062 +0000 UTC m=+948.956587104" watchObservedRunningTime="2026-11-03 14:27:53.399304654 +0000 UTC m=+948.957941138" Nov 03 14:27:53.443045 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.442992 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6"] Nov 03 14:27:53.453845 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.453809 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-l88r6"] Nov 03 14:27:53.587760 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:53.587723 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:27:54.219709 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:54.219671 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr"] Nov 03 14:27:54.619145 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:54.619059 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65281ce4-925e-4e47-82a7-850296c07953" path="/var/lib/kubelet/pods/65281ce4-925e-4e47-82a7-850296c07953/volumes" Nov 03 14:27:58.189455 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.189394 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" podUID="e94c5fe4-b016-471c-bc6d-410c843851f0" containerName="payload-processing" containerID="cri-o://59a0b5fb02946d621805bdf7d8166a0b76750143f4a25d3cf1b325f2507db012" gracePeriod=25 Nov 03 14:27:58.378570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.378537 6389 generic.go:356] "Generic (PLEG): container finished" podID="e94c5fe4-b016-471c-bc6d-410c843851f0" containerID="59a0b5fb02946d621805bdf7d8166a0b76750143f4a25d3cf1b325f2507db012" exitCode=0 Nov 03 14:27:58.378734 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.378681 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" event={"ID":"e94c5fe4-b016-471c-bc6d-410c843851f0","Type":"ContainerDied","Data":"59a0b5fb02946d621805bdf7d8166a0b76750143f4a25d3cf1b325f2507db012"} Nov 03 14:27:58.384018 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:27:58.383986 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:27:58.457500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.457479 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:58.548169 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.548139 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-kube-api-access-dvfr4\" (UniqueName: \"kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-kube-api-access-dvfr4\") pod \"e94c5fe4-b016-471c-bc6d-410c843851f0\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " Nov 03 14:27:58.548327 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.548185 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-routing-overlay-volume\") pod \"e94c5fe4-b016-471c-bc6d-410c843851f0\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " Nov 03 14:27:58.548327 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.548231 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-plugins-config-volume\") pod \"e94c5fe4-b016-471c-bc6d-410c843851f0\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " Nov 03 14:27:58.548327 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.548274 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-provider-credentials\") pod \"e94c5fe4-b016-471c-bc6d-410c843851f0\" (UID: \"e94c5fe4-b016-471c-bc6d-410c843851f0\") " Nov 03 14:27:58.548665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.548602 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-routing-overlay-volume" pod "e94c5fe4-b016-471c-bc6d-410c843851f0" (UID: "e94c5fe4-b016-471c-bc6d-410c843851f0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:58.548795 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.548679 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-plugins-config-volume" pod "e94c5fe4-b016-471c-bc6d-410c843851f0" (UID: "e94c5fe4-b016-471c-bc6d-410c843851f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:58.550845 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.550817 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-kube-api-access-dvfr4" pod "e94c5fe4-b016-471c-bc6d-410c843851f0" (UID: "e94c5fe4-b016-471c-bc6d-410c843851f0"). InnerVolumeSpecName "kube-api-access-dvfr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:58.550967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.550893 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-provider-credentials" pod "e94c5fe4-b016-471c-bc6d-410c843851f0" (UID: "e94c5fe4-b016-471c-bc6d-410c843851f0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:58.649379 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.649350 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvfr4\" (UniqueName: \"kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-kube-api-access-dvfr4\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:58.649379 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.649374 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:58.649379 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.649385 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94c5fe4-b016-471c-bc6d-410c843851f0-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:58.649714 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.649394 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e94c5fe4-b016-471c-bc6d-410c843851f0-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:27:58.722376 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:58.722285 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858"] Nov 03 14:27:59.220691 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:59.220629 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" podUID="6c04787d-8f27-4b2e-bc09-1066c7d38920" containerName="payload-processing" containerID="cri-o://6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24" gracePeriod=25 Nov 03 14:27:59.386887 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:59.386859 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" Nov 03 14:27:59.386887 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:59.386868 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5" event={"ID":"e94c5fe4-b016-471c-bc6d-410c843851f0","Type":"ContainerDied","Data":"ea3297c957ae721acc2ca69148fc0a340451d7afd96688dd6a2d0f5ffc0f3a7c"} Nov 03 14:27:59.387126 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:59.386946 6389 scope.go:122] "RemoveContainer" containerID="59a0b5fb02946d621805bdf7d8166a0b76750143f4a25d3cf1b325f2507db012" Nov 03 14:27:59.443661 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:59.443630 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5"] Nov 03 14:27:59.456424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:27:59.456398 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-f9fcdzwg5"] Nov 03 14:28:00.136299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.136258 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9"] Nov 03 14:28:00.137070 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.137049 6389 cpu_manager.go:414] "containerMap: removing container" podUID="e94c5fe4-b016-471c-bc6d-410c843851f0" containerName="payload-processing" Nov 03 14:28:00.137132 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.137074 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94c5fe4-b016-471c-bc6d-410c843851f0" containerName="payload-processing" Nov 03 14:28:00.137132 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.137098 6389 cpu_manager.go:414] "containerMap: removing container" podUID="65281ce4-925e-4e47-82a7-850296c07953" containerName="istio-proxy" Nov 03 14:28:00.137132 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.137105 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65281ce4-925e-4e47-82a7-850296c07953" containerName="istio-proxy" Nov 03 14:28:00.137370 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.137351 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94c5fe4-b016-471c-bc6d-410c843851f0" containerName="payload-processing" Nov 03 14:28:00.137370 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.137372 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="65281ce4-925e-4e47-82a7-850296c07953" containerName="istio-proxy" Nov 03 14:28:00.169644 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.169545 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.186114 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.186070 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9"] Nov 03 14:28:00.189495 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.187988 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2d0da2-openshift-default-dockercfg-288bc\"" Nov 03 14:28:00.271725 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.271696 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.272077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.271767 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.272077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.271853 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.272077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.271897 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9ks4\" (UniqueName: \"kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-kube-api-access-m9ks4\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.272077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.271958 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.272077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.272001 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4464194f-c9e3-4bfa-951c-1b8897ef028c-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.272251 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.272101 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.272251 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.272118 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.272251 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.272174 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373179 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373142 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373179 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373183 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373230 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373249 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373298 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373326 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373357 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9ks4\" (UniqueName: \"kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-kube-api-access-m9ks4\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373379 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373416 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4464194f-c9e3-4bfa-951c-1b8897ef028c-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373591 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373700 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373917 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373864 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.373981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.373958 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.374218 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.374198 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4464194f-c9e3-4bfa-951c-1b8897ef028c-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.375687 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.375666 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.376140 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.376122 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.394504 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.394437 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.394627 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.394558 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9ks4\" (UniqueName: \"kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-kube-api-access-m9ks4\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.526405 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.526376 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:00.589950 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.587059 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:28:00.608504 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.608222 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:28:00.620644 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.619543 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94c5fe4-b016-471c-bc6d-410c843851f0" path="/var/lib/kubelet/pods/e94c5fe4-b016-471c-bc6d-410c843851f0/volumes" Nov 03 14:28:00.846130 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.846107 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9"] Nov 03 14:28:00.847648 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:28:00.847622 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4464194f_c9e3_4bfa_951c_1b8897ef028c.slice/crio-b0576dae4e8a4062dff4ca6dfa8498c446ff72181e27e4700394a35f8a8e3525 WatchSource:0}: Error finding container b0576dae4e8a4062dff4ca6dfa8498c446ff72181e27e4700394a35f8a8e3525: Status 404 returned error can't find the container with id b0576dae4e8a4062dff4ca6dfa8498c446ff72181e27e4700394a35f8a8e3525 Nov 03 14:28:00.850073 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.850038 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:28:00.850154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.850111 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:28:00.850154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:00.850143 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:28:01.403452 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:01.403415 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" event={"ID":"4464194f-c9e3-4bfa-951c-1b8897ef028c","Type":"ContainerStarted","Data":"4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0"} Nov 03 14:28:01.403934 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:01.403483 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" event={"ID":"4464194f-c9e3-4bfa-951c-1b8897ef028c","Type":"ContainerStarted","Data":"b0576dae4e8a4062dff4ca6dfa8498c446ff72181e27e4700394a35f8a8e3525"} Nov 03 14:28:01.415010 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:01.414986 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:28:01.467818 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:01.467764 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" podStartSLOduration=1.467748848 podStartE2EDuration="1.467748848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:01.46633874 +0000 UTC m=+957.024975223" watchObservedRunningTime="2026-11-03 14:28:01.467748848 +0000 UTC m=+957.026385331" Nov 03 14:28:01.527106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:01.527058 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:01.528500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:01.528470 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:28:01.528671 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:01.528527 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 14:28:02.532284 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:02.532235 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:28:02.532984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:02.532949 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 14:28:03.541639 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:03.529997 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:28:03.541639 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:03.530071 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 14:28:03.734641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:03.724701 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" podUID="92f3dd10-e08b-450b-b2f2-531d97c28d77" containerName="payload-processing" containerID="cri-o://1bcd9e6d38979887b93a4ada878ec219a20b74400f9d6c464963b99e0deea206" gracePeriod=25 Nov 03 14:28:03.977634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:03.959933 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg"] Nov 03 14:28:03.978102 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:03.978073 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" podUID="e6180463-5ac5-4422-9b30-d60723c15deb" containerName="maas-api" containerID="cri-o://5592f41795ed9cf6d8d82d276dd5a8a68692ba3fc7d2c2b40516b496c8e788dc" gracePeriod=30 Nov 03 14:28:04.571651 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:04.567887 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:28:04.571651 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:04.567941 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 14:28:05.341011 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.340971 6389 generic.go:356] "Generic (PLEG): container finished" podID="92f3dd10-e08b-450b-b2f2-531d97c28d77" containerID="1bcd9e6d38979887b93a4ada878ec219a20b74400f9d6c464963b99e0deea206" exitCode=0 Nov 03 14:28:05.341459 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.341433 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" event={"ID":"92f3dd10-e08b-450b-b2f2-531d97c28d77","Type":"ContainerDied","Data":"1bcd9e6d38979887b93a4ada878ec219a20b74400f9d6c464963b99e0deea206"} Nov 03 14:28:05.348816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.348790 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:28:05.415687 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.410828 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" podUID="6c04787d-8f27-4b2e-bc09-1066c7d38920" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:05.513297 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.513253 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-routing-overlay-volume\") pod \"92f3dd10-e08b-450b-b2f2-531d97c28d77\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " Nov 03 14:28:05.513757 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.513718 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-provider-credentials\") pod \"92f3dd10-e08b-450b-b2f2-531d97c28d77\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " Nov 03 14:28:05.514057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.514023 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-plugins-config-volume\") pod \"92f3dd10-e08b-450b-b2f2-531d97c28d77\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " Nov 03 14:28:05.514218 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.514201 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-kube-api-access-xml2r\" (UniqueName: \"kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-kube-api-access-xml2r\") pod \"92f3dd10-e08b-450b-b2f2-531d97c28d77\" (UID: \"92f3dd10-e08b-450b-b2f2-531d97c28d77\") " Nov 03 14:28:05.608634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.583291 6389 generic.go:356] "Generic (PLEG): container finished" podID="e6180463-5ac5-4422-9b30-d60723c15deb" containerID="5592f41795ed9cf6d8d82d276dd5a8a68692ba3fc7d2c2b40516b496c8e788dc" exitCode=0 Nov 03 14:28:05.609316 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.609276 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" event={"ID":"e6180463-5ac5-4422-9b30-d60723c15deb","Type":"ContainerDied","Data":"5592f41795ed9cf6d8d82d276dd5a8a68692ba3fc7d2c2b40516b496c8e788dc"} Nov 03 14:28:05.694635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.689210 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:28:05.694635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.689270 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 14:28:05.694635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.690350 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-routing-overlay-volume" pod "92f3dd10-e08b-450b-b2f2-531d97c28d77" (UID: "92f3dd10-e08b-450b-b2f2-531d97c28d77"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:05.719093 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.719054 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-plugins-config-volume" pod "92f3dd10-e08b-450b-b2f2-531d97c28d77" (UID: "92f3dd10-e08b-450b-b2f2-531d97c28d77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:05.743677 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.742858 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:05.743677 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.742893 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f3dd10-e08b-450b-b2f2-531d97c28d77-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:05.827870 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.825028 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:28:05.828205 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.828173 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-kube-api-access-xml2r" pod "92f3dd10-e08b-450b-b2f2-531d97c28d77" (UID: "92f3dd10-e08b-450b-b2f2-531d97c28d77"). InnerVolumeSpecName "kube-api-access-xml2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:05.828755 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.828721 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-provider-credentials" pod "92f3dd10-e08b-450b-b2f2-531d97c28d77" (UID: "92f3dd10-e08b-450b-b2f2-531d97c28d77"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:05.860826 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.852879 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:05.860826 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.852907 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xml2r\" (UniqueName: \"kubernetes.io/projected/92f3dd10-e08b-450b-b2f2-531d97c28d77-kube-api-access-xml2r\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:05.972212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.969003 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-maas-api-tls\") pod \"e6180463-5ac5-4422-9b30-d60723c15deb\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " Nov 03 14:28:05.972212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.969106 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6180463-5ac5-4422-9b30-d60723c15deb-kube-api-access-4bpc5\" (UniqueName: \"kubernetes.io/projected/e6180463-5ac5-4422-9b30-d60723c15deb-kube-api-access-4bpc5\") pod \"e6180463-5ac5-4422-9b30-d60723c15deb\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " Nov 03 14:28:05.972212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.969163 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6180463-5ac5-4422-9b30-d60723c15deb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6180463-5ac5-4422-9b30-d60723c15deb-openshift-service-ca\") pod \"e6180463-5ac5-4422-9b30-d60723c15deb\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " Nov 03 14:28:05.972212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.969273 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-metrics-certs\") pod \"e6180463-5ac5-4422-9b30-d60723c15deb\" (UID: \"e6180463-5ac5-4422-9b30-d60723c15deb\") " Nov 03 14:28:05.987374 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:05.985569 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-metrics-certs" pod "e6180463-5ac5-4422-9b30-d60723c15deb" (UID: "e6180463-5ac5-4422-9b30-d60723c15deb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:06.014343 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.014287 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6180463-5ac5-4422-9b30-d60723c15deb-openshift-service-ca" pod "e6180463-5ac5-4422-9b30-d60723c15deb" (UID: "e6180463-5ac5-4422-9b30-d60723c15deb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:06.171635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.091364 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6180463-5ac5-4422-9b30-d60723c15deb-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:06.171635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.091401 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:06.171635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.091570 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6180463-5ac5-4422-9b30-d60723c15deb-kube-api-access-4bpc5" pod "e6180463-5ac5-4422-9b30-d60723c15deb" (UID: "e6180463-5ac5-4422-9b30-d60723c15deb"). InnerVolumeSpecName "kube-api-access-4bpc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:06.241744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.197358 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bpc5\" (UniqueName: \"kubernetes.io/projected/e6180463-5ac5-4422-9b30-d60723c15deb-kube-api-access-4bpc5\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:06.242573 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.242507 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-maas-api-tls" pod "e6180463-5ac5-4422-9b30-d60723c15deb" (UID: "e6180463-5ac5-4422-9b30-d60723c15deb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:06.351659 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.345481 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6180463-5ac5-4422-9b30-d60723c15deb-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:06.548719 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.548195 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:28:06.548719 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.548276 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 14:28:06.686275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.686243 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" Nov 03 14:28:06.689263 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.689222 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858" event={"ID":"92f3dd10-e08b-450b-b2f2-531d97c28d77","Type":"ContainerDied","Data":"d4083ece55ac4eb7c5f112ea0662fb61baeb73a181af3c0ad284ceae787097c6"} Nov 03 14:28:06.689372 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.689300 6389 scope.go:122] "RemoveContainer" containerID="1bcd9e6d38979887b93a4ada878ec219a20b74400f9d6c464963b99e0deea206" Nov 03 14:28:06.775501 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.775392 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" event={"ID":"e6180463-5ac5-4422-9b30-d60723c15deb","Type":"ContainerDied","Data":"d93f804cd389b5f54880cebaeec401ea611211e705cb5b6b50de0c81e28b8def"} Nov 03 14:28:06.775662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.775515 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg" Nov 03 14:28:06.850629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.848830 6389 scope.go:122] "RemoveContainer" containerID="5592f41795ed9cf6d8d82d276dd5a8a68692ba3fc7d2c2b40516b496c8e788dc" Nov 03 14:28:06.903709 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.903663 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858"] Nov 03 14:28:06.925651 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.905993 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b9b2858"] Nov 03 14:28:06.967571 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.967540 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg"] Nov 03 14:28:06.992753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:06.992729 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e759e5-5bb4b5c4f5-rxnbg"] Nov 03 14:28:07.617750 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:07.617714 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:28:07.618005 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:07.617977 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 14:28:08.529109 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:08.529068 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:28:08.532731 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:08.532686 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 14:28:08.682718 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:08.682683 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f3dd10-e08b-450b-b2f2-531d97c28d77" path="/var/lib/kubelet/pods/92f3dd10-e08b-450b-b2f2-531d97c28d77/volumes" Nov 03 14:28:08.683600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:08.683575 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6180463-5ac5-4422-9b30-d60723c15deb" path="/var/lib/kubelet/pods/e6180463-5ac5-4422-9b30-d60723c15deb/volumes" Nov 03 14:28:09.775152 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:09.775107 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:09.775696 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:09.775484 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:09.776389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:09.776371 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:28:13.581911 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:13.581881 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482"] Nov 03 14:28:13.582547 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:13.582135 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" podUID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" containerName="istio-proxy" containerID="cri-o://7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414" gracePeriod=30 Nov 03 14:28:15.262905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:15.262863 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" podUID="6c04787d-8f27-4b2e-bc09-1066c7d38920" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:17.020218 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.020162 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp"] Nov 03 14:28:17.021006 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.020979 6389 cpu_manager.go:414] "containerMap: removing container" podUID="e6180463-5ac5-4422-9b30-d60723c15deb" containerName="maas-api" Nov 03 14:28:17.021006 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.021004 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6180463-5ac5-4422-9b30-d60723c15deb" containerName="maas-api" Nov 03 14:28:17.021189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.021060 6389 cpu_manager.go:414] "containerMap: removing container" podUID="92f3dd10-e08b-450b-b2f2-531d97c28d77" containerName="payload-processing" Nov 03 14:28:17.021189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.021069 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f3dd10-e08b-450b-b2f2-531d97c28d77" containerName="payload-processing" Nov 03 14:28:17.021282 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.021203 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f3dd10-e08b-450b-b2f2-531d97c28d77" containerName="payload-processing" Nov 03 14:28:17.021282 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.021220 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6180463-5ac5-4422-9b30-d60723c15deb" containerName="maas-api" Nov 03 14:28:17.027126 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.027100 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.059665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.059596 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b4361f-openshift-default-dockercfg-h8cr5\"" Nov 03 14:28:17.062273 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.062242 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp"] Nov 03 14:28:17.075751 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.075719 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.075926 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.075807 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.075926 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.075834 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.075926 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.075878 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6871153-f93d-4cc0-b09b-f8f303706858-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.075926 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.075920 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6871153-f93d-4cc0-b09b-f8f303706858-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.076154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.075943 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.076154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.075996 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.076154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.076037 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87f9k\" (UniqueName: \"kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-kube-api-access-87f9k\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.076154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.076090 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.177261 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.177230 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.177433 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.177280 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87f9k\" (UniqueName: \"kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-kube-api-access-87f9k\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.177433 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.177342 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.177511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.177491 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.177580 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.177561 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.177668 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.177599 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.177734 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.177682 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6871153-f93d-4cc0-b09b-f8f303706858-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.177734 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.177719 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.177844 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.177737 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.177844 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.177741 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6871153-f93d-4cc0-b09b-f8f303706858-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.177844 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.177799 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.178003 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.177953 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.178120 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.178095 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.178313 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.178296 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6871153-f93d-4cc0-b09b-f8f303706858-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.180099 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.180070 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.180346 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.180328 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6871153-f93d-4cc0-b09b-f8f303706858-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.205086 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.205052 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.205277 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.205257 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87f9k\" (UniqueName: \"kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-kube-api-access-87f9k\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.340913 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.340824 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:17.699486 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.699455 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp"] Nov 03 14:28:17.711668 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:28:17.711642 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6871153_f93d_4cc0_b09b_f8f303706858.slice/crio-f5365f9dba5ee67515d68326b7220d6ecedd0f3cd9c582ac331ac5e526c3b1a6 WatchSource:0}: Error finding container f5365f9dba5ee67515d68326b7220d6ecedd0f3cd9c582ac331ac5e526c3b1a6: Status 404 returned error can't find the container with id f5365f9dba5ee67515d68326b7220d6ecedd0f3cd9c582ac331ac5e526c3b1a6 Nov 03 14:28:17.714080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.714051 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:28:17.714147 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.714111 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:28:17.714147 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:17.714143 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:28:18.101108 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:18.101024 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" event={"ID":"e6871153-f93d-4cc0-b09b-f8f303706858","Type":"ContainerStarted","Data":"a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400"} Nov 03 14:28:18.101108 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:18.101086 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" event={"ID":"e6871153-f93d-4cc0-b09b-f8f303706858","Type":"ContainerStarted","Data":"f5365f9dba5ee67515d68326b7220d6ecedd0f3cd9c582ac331ac5e526c3b1a6"} Nov 03 14:28:18.161647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:18.161574 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" podStartSLOduration=2.16155901 podStartE2EDuration="2.16155901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:18.160159127 +0000 UTC m=+973.718795611" watchObservedRunningTime="2026-11-03 14:28:18.16155901 +0000 UTC m=+973.720195495" Nov 03 14:28:18.313572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:18.313535 6389 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:28:18.313765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:18.313620 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" podUID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:28:18.342289 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:18.342236 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:18.343756 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:18.343725 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 14:28:18.344040 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:18.344007 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 14:28:18.916040 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:18.916017 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:28:19.005352 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.005315 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-envoy\") pod \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " Nov 03 14:28:19.005525 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.005369 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-credential-socket\") pod \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " Nov 03 14:28:19.005525 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.005443 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-token\" (UniqueName: \"kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-token\") pod \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " Nov 03 14:28:19.005525 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.005478 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-certs\") pod \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " Nov 03 14:28:19.005525 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.005508 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-kube-api-access-blfd7\" (UniqueName: \"kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-kube-api-access-blfd7\") pod \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " Nov 03 14:28:19.005816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.005558 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-podinfo\") pod \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " Nov 03 14:28:19.005816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.005727 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-socket\") pod \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " Nov 03 14:28:19.005816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.005812 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istiod-ca-cert\") pod \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " Nov 03 14:28:19.005970 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.005890 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-data\") pod \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\" (UID: \"6b1bf405-ee22-4a6d-9ff6-f0651334075e\") " Nov 03 14:28:19.006622 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.006581 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-data" pod "6b1bf405-ee22-4a6d-9ff6-f0651334075e" (UID: "6b1bf405-ee22-4a6d-9ff6-f0651334075e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:19.006942 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.006829 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-certs" pod "6b1bf405-ee22-4a6d-9ff6-f0651334075e" (UID: "6b1bf405-ee22-4a6d-9ff6-f0651334075e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:19.007033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.006969 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-socket" pod "6b1bf405-ee22-4a6d-9ff6-f0651334075e" (UID: "6b1bf405-ee22-4a6d-9ff6-f0651334075e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:19.007401 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.007376 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istiod-ca-cert" pod "6b1bf405-ee22-4a6d-9ff6-f0651334075e" (UID: "6b1bf405-ee22-4a6d-9ff6-f0651334075e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:19.007702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.007681 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-credential-socket" pod "6b1bf405-ee22-4a6d-9ff6-f0651334075e" (UID: "6b1bf405-ee22-4a6d-9ff6-f0651334075e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:19.008451 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.008401 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-envoy" pod "6b1bf405-ee22-4a6d-9ff6-f0651334075e" (UID: "6b1bf405-ee22-4a6d-9ff6-f0651334075e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:19.009204 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.009146 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-podinfo" pod "6b1bf405-ee22-4a6d-9ff6-f0651334075e" (UID: "6b1bf405-ee22-4a6d-9ff6-f0651334075e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:28:19.010907 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.010882 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-kube-api-access-blfd7" pod "6b1bf405-ee22-4a6d-9ff6-f0651334075e" (UID: "6b1bf405-ee22-4a6d-9ff6-f0651334075e"). InnerVolumeSpecName "kube-api-access-blfd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:19.013975 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.013948 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-token" pod "6b1bf405-ee22-4a6d-9ff6-f0651334075e" (UID: "6b1bf405-ee22-4a6d-9ff6-f0651334075e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:19.107228 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107202 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.107228 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107227 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.107726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107241 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.107726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107255 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.107726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107266 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blfd7\" (UniqueName: \"kubernetes.io/projected/6b1bf405-ee22-4a6d-9ff6-f0651334075e-kube-api-access-blfd7\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.107726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107278 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.107726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107290 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.107726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107302 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.107726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107315 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b1bf405-ee22-4a6d-9ff6-f0651334075e-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.107726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107457 6389 generic.go:356] "Generic (PLEG): container finished" podID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" containerID="7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414" exitCode=0 Nov 03 14:28:19.107726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107529 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" Nov 03 14:28:19.107726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107550 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" event={"ID":"6b1bf405-ee22-4a6d-9ff6-f0651334075e","Type":"ContainerDied","Data":"7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414"} Nov 03 14:28:19.107726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107600 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482" event={"ID":"6b1bf405-ee22-4a6d-9ff6-f0651334075e","Type":"ContainerDied","Data":"1cf02a989239b2c2b7a5e35032da70e26763d791adac6ad68d69abc0bc90c043"} Nov 03 14:28:19.107726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.107651 6389 scope.go:122] "RemoveContainer" containerID="7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414" Nov 03 14:28:19.156673 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.156647 6389 scope.go:122] "RemoveContainer" containerID="7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414" Nov 03 14:28:19.157046 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:28:19.157029 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414\": container with ID starting with 7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414 not found: ID does not exist" containerID="7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414" Nov 03 14:28:19.157113 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.157055 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414"} err="failed to get container status \"7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414\": rpc error: code = NotFound desc = could not find container \"7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414\": container with ID starting with 7b937f194788042727657830a7699f7559f40854c08180bae5ff26c44d72c414 not found: ID does not exist" Nov 03 14:28:19.194474 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.194446 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482"] Nov 03 14:28:19.202166 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.202138 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-d8482"] Nov 03 14:28:19.347691 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.347151 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 14:28:19.347691 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:19.347224 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 14:28:20.216639 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.216477 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" Nov 03 14:28:20.217135 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.217029 6389 generic.go:356] "Generic (PLEG): container finished" podID="6c04787d-8f27-4b2e-bc09-1066c7d38920" containerID="6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24" exitCode=0 Nov 03 14:28:20.217208 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.217134 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" event={"ID":"6c04787d-8f27-4b2e-bc09-1066c7d38920","Type":"ContainerDied","Data":"6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24"} Nov 03 14:28:20.217268 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.217215 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" event={"ID":"6c04787d-8f27-4b2e-bc09-1066c7d38920","Type":"ContainerDied","Data":"f7fced8c10b1c540295239791556b85461625a9dfc4c34cf2c4e9f02d56d1dcf"} Nov 03 14:28:20.217268 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.217248 6389 scope.go:122] "RemoveContainer" containerID="6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24" Nov 03 14:28:20.262743 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.262703 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c04787d-8f27-4b2e-bc09-1066c7d38920-kube-api-access-jh8q5\" (UniqueName: \"kubernetes.io/projected/6c04787d-8f27-4b2e-bc09-1066c7d38920-kube-api-access-jh8q5\") pod \"6c04787d-8f27-4b2e-bc09-1066c7d38920\" (UID: \"6c04787d-8f27-4b2e-bc09-1066c7d38920\") " Nov 03 14:28:20.263483 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.263460 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c04787d-8f27-4b2e-bc09-1066c7d38920-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c04787d-8f27-4b2e-bc09-1066c7d38920-plugins-config-volume\") pod \"6c04787d-8f27-4b2e-bc09-1066c7d38920\" (UID: \"6c04787d-8f27-4b2e-bc09-1066c7d38920\") " Nov 03 14:28:20.264153 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.264050 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c04787d-8f27-4b2e-bc09-1066c7d38920-plugins-config-volume" pod "6c04787d-8f27-4b2e-bc09-1066c7d38920" (UID: "6c04787d-8f27-4b2e-bc09-1066c7d38920"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:20.266209 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.266192 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c04787d-8f27-4b2e-bc09-1066c7d38920-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:20.360144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.358594 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 14:28:20.360144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.358677 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 14:28:20.439009 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.438365 6389 scope.go:122] "RemoveContainer" containerID="6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24" Nov 03 14:28:20.449203 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:28:20.448942 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24\": container with ID starting with 6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24 not found: ID does not exist" containerID="6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24" Nov 03 14:28:20.449203 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.448992 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24"} err="failed to get container status \"6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24\": rpc error: code = NotFound desc = could not find container \"6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24\": container with ID starting with 6805f4c6e147c41dbbab68f2993c86739b4dab94bef6a865b65b36fa624d5f24 not found: ID does not exist" Nov 03 14:28:20.540963 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.540908 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c04787d-8f27-4b2e-bc09-1066c7d38920-kube-api-access-jh8q5" pod "6c04787d-8f27-4b2e-bc09-1066c7d38920" (UID: "6c04787d-8f27-4b2e-bc09-1066c7d38920"). InnerVolumeSpecName "kube-api-access-jh8q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:20.598212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.598167 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh8q5\" (UniqueName: \"kubernetes.io/projected/6c04787d-8f27-4b2e-bc09-1066c7d38920-kube-api-access-jh8q5\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:20.870529 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:20.870488 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" path="/var/lib/kubelet/pods/6b1bf405-ee22-4a6d-9ff6-f0651334075e/volumes" Nov 03 14:28:21.273298 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:21.273267 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr" Nov 03 14:28:21.367674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:21.367635 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 14:28:21.367954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:21.367926 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 14:28:21.424397 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:21.424348 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" podUID="151733fd-8ae3-4401-870c-8384e18d768b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:28:21.489634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:21.456730 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr"] Nov 03 14:28:21.506668 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:21.491956 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e759e5-57578fb859-jjjsr"] Nov 03 14:28:22.244743 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:22.244692 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" podUID="76e983d7-ca69-44d1-857c-65eb03ec5e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:28:22.355598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:22.355132 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 14:28:22.355598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:22.355191 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 14:28:22.497692 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:22.496597 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" podUID="151733fd-8ae3-4401-870c-8384e18d768b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:28:22.497692 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:22.497347 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:28:23.201402 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:23.201360 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c04787d-8f27-4b2e-bc09-1066c7d38920" path="/var/lib/kubelet/pods/6c04787d-8f27-4b2e-bc09-1066c7d38920/volumes" Nov 03 14:28:23.296791 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:23.296762 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:28:23.492432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:23.492120 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 14:28:23.492991 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:23.492960 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 14:28:24.344038 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:24.344006 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 14:28:24.344299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:24.344265 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 14:28:25.342984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:25.342945 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 14:28:25.343544 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:25.343512 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 14:28:26.346215 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.346163 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:26.346781 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.346747 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:26.347732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.347708 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:28:26.588804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.588767 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk"] Nov 03 14:28:26.589681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.589657 6389 cpu_manager.go:414] "containerMap: removing container" podUID="6c04787d-8f27-4b2e-bc09-1066c7d38920" containerName="payload-processing" Nov 03 14:28:26.589865 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.589850 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c04787d-8f27-4b2e-bc09-1066c7d38920" containerName="payload-processing" Nov 03 14:28:26.590045 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.590033 6389 cpu_manager.go:414] "containerMap: removing container" podUID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" containerName="istio-proxy" Nov 03 14:28:26.590131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.590122 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" containerName="istio-proxy" Nov 03 14:28:26.590393 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.590380 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b1bf405-ee22-4a6d-9ff6-f0651334075e" containerName="istio-proxy" Nov 03 14:28:26.590513 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.590475 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c04787d-8f27-4b2e-bc09-1066c7d38920" containerName="payload-processing" Nov 03 14:28:26.601069 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.601007 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk"] Nov 03 14:28:26.601406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.601323 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.653641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.648358 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-podinfo\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.653641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.648391 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istiod-ca-cert\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.653641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.648455 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-socket\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.653641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.648531 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4972\" (UniqueName: \"kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-kube-api-access-h4972\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.653641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.648786 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-envoy\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.653641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.648923 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-token\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.653641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.648951 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-data\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.653641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.649026 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-certs\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.653641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.649091 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-credential-socket\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.677764 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.677739 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cbd38938-openshift-default-dockercfg-6sjjj\"" Nov 03 14:28:26.749861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.749822 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-token\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.749861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.749864 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-data\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.750127 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.749937 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-certs\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.750127 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.749967 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-credential-socket\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.750127 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.750015 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-podinfo\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.750127 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.750041 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istiod-ca-cert\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.750127 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.750110 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-socket\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.750396 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.750138 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4972\" (UniqueName: \"kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-kube-api-access-h4972\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.750396 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.750338 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-envoy\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.750515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.750403 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-certs\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.750515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.750412 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-credential-socket\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.750780 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.750757 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-data\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.750912 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.750887 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-socket\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.751071 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.750905 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istiod-ca-cert\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.752744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.752723 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-podinfo\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.752858 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.752826 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-envoy\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.773065 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.773033 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-token\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.773228 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.773179 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4972\" (UniqueName: \"kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-kube-api-access-h4972\") pod \"e2e-ait-cbd38938-openshift-default-55744f6757-mdddk\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:26.916883 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:26.916852 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:27.236100 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:27.236010 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk"] Nov 03 14:28:27.239670 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:28:27.239644 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba32398_976e_4f2d_9cf1_b00f04bb56ff.slice/crio-e6e8df9f336ccd667fab6f46df2d6edd78f7d90952f4daa38576a4316fc9f1f0 WatchSource:0}: Error finding container e6e8df9f336ccd667fab6f46df2d6edd78f7d90952f4daa38576a4316fc9f1f0: Status 404 returned error can't find the container with id e6e8df9f336ccd667fab6f46df2d6edd78f7d90952f4daa38576a4316fc9f1f0 Nov 03 14:28:27.242055 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:27.242029 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:28:27.242128 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:27.242094 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:28:27.242128 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:27.242120 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:28:27.873119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:27.873085 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" event={"ID":"5ba32398-976e-4f2d-9cf1-b00f04bb56ff","Type":"ContainerStarted","Data":"f583d8d261453361f0ae1e1e8c9b573ce24c4fa8733daf8f14b2acc5a494f627"} Nov 03 14:28:27.873541 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:27.873139 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" event={"ID":"5ba32398-976e-4f2d-9cf1-b00f04bb56ff","Type":"ContainerStarted","Data":"e6e8df9f336ccd667fab6f46df2d6edd78f7d90952f4daa38576a4316fc9f1f0"} Nov 03 14:28:27.917755 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:27.917731 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:27.932948 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:27.932893 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" podStartSLOduration=1.932876823 podStartE2EDuration="1.932876823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:27.932166501 +0000 UTC m=+983.490802987" watchObservedRunningTime="2026-11-03 14:28:27.932876823 +0000 UTC m=+983.491513307" Nov 03 14:28:28.375061 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:28:28.375026 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:28:28.922334 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:28.922311 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:29.884828 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:29.884796 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:29.886572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:29.886549 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:28:31.617573 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.617538 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw"] Nov 03 14:28:31.622345 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.622319 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" Nov 03 14:28:31.641009 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.640982 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw"] Nov 03 14:28:31.653896 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.653865 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cbd38938\"" Nov 03 14:28:31.653896 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.653875 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cbd38938-dockercfg-l5tr6\"" Nov 03 14:28:31.735156 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.734798 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7rpp\" (UniqueName: \"kubernetes.io/projected/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-kube-api-access-w7rpp\") pod \"payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw\" (UID: \"63a0d285-a6a2-455c-afb2-625e5a8dd0ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" Nov 03 14:28:31.735156 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.734949 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw\" (UID: \"63a0d285-a6a2-455c-afb2-625e5a8dd0ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" Nov 03 14:28:31.742916 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.742870 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc"] Nov 03 14:28:31.757049 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.757019 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" Nov 03 14:28:31.838533 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.836077 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw\" (UID: \"63a0d285-a6a2-455c-afb2-625e5a8dd0ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" Nov 03 14:28:31.838533 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.836213 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-plugins-config-volume\") pod \"payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc\" (UID: \"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd\") " pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" Nov 03 14:28:31.838533 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.836279 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7rpp\" (UniqueName: \"kubernetes.io/projected/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-kube-api-access-w7rpp\") pod \"payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw\" (UID: \"63a0d285-a6a2-455c-afb2-625e5a8dd0ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" Nov 03 14:28:31.838533 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.836323 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7fpk\" (UniqueName: \"kubernetes.io/projected/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-kube-api-access-w7fpk\") pod \"payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc\" (UID: \"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd\") " pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" Nov 03 14:28:31.838533 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.837256 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw\" (UID: \"63a0d285-a6a2-455c-afb2-625e5a8dd0ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" Nov 03 14:28:31.838533 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.838074 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc"] Nov 03 14:28:31.863934 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.863906 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7rpp\" (UniqueName: \"kubernetes.io/projected/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-kube-api-access-w7rpp\") pod \"payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw\" (UID: \"63a0d285-a6a2-455c-afb2-625e5a8dd0ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" Nov 03 14:28:31.937535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.937502 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7fpk\" (UniqueName: \"kubernetes.io/projected/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-kube-api-access-w7fpk\") pod \"payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc\" (UID: \"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd\") " pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" Nov 03 14:28:31.937775 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.937761 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-plugins-config-volume\") pod \"payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc\" (UID: \"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd\") " pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" Nov 03 14:28:31.938407 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.938388 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-plugins-config-volume\") pod \"payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc\" (UID: \"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd\") " pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" Nov 03 14:28:31.941258 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.941243 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" Nov 03 14:28:31.959890 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:31.959868 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7fpk\" (UniqueName: \"kubernetes.io/projected/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-kube-api-access-w7fpk\") pod \"payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc\" (UID: \"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd\") " pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" Nov 03 14:28:32.155463 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:32.155423 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" Nov 03 14:28:32.257807 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:32.257655 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw"] Nov 03 14:28:32.259057 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:28:32.259007 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a0d285_a6a2_455c_afb2_625e5a8dd0ef.slice/crio-98aa93b9cbf95d7029493d2dc1a19dec023e696f56a05488182cc4036b867589 WatchSource:0}: Error finding container 98aa93b9cbf95d7029493d2dc1a19dec023e696f56a05488182cc4036b867589: Status 404 returned error can't find the container with id 98aa93b9cbf95d7029493d2dc1a19dec023e696f56a05488182cc4036b867589 Nov 03 14:28:32.471104 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:32.471077 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc"] Nov 03 14:28:32.472045 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:28:32.472015 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b51c8bc_90ac_4257_8c51_dbcd40ac34bd.slice/crio-d9ebc3187f8540e58a87828ee878aa8bd7540f984ac65aedc76aca6cd66de6db WatchSource:0}: Error finding container d9ebc3187f8540e58a87828ee878aa8bd7540f984ac65aedc76aca6cd66de6db: Status 404 returned error can't find the container with id d9ebc3187f8540e58a87828ee878aa8bd7540f984ac65aedc76aca6cd66de6db Nov 03 14:28:32.901446 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:32.901365 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" event={"ID":"63a0d285-a6a2-455c-afb2-625e5a8dd0ef","Type":"ContainerStarted","Data":"31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14"} Nov 03 14:28:32.901446 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:32.901423 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" event={"ID":"63a0d285-a6a2-455c-afb2-625e5a8dd0ef","Type":"ContainerStarted","Data":"98aa93b9cbf95d7029493d2dc1a19dec023e696f56a05488182cc4036b867589"} Nov 03 14:28:32.901917 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:32.901505 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" Nov 03 14:28:32.902987 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:32.902964 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" event={"ID":"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd","Type":"ContainerStarted","Data":"b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e"} Nov 03 14:28:32.903107 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:32.903003 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" event={"ID":"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd","Type":"ContainerStarted","Data":"d9ebc3187f8540e58a87828ee878aa8bd7540f984ac65aedc76aca6cd66de6db"} Nov 03 14:28:32.903107 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:32.903023 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" Nov 03 14:28:32.947768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:32.947718 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" podStartSLOduration=1.947703977 podStartE2EDuration="1.947703977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:32.944937462 +0000 UTC m=+988.503573946" watchObservedRunningTime="2026-11-03 14:28:32.947703977 +0000 UTC m=+988.506340461" Nov 03 14:28:32.990573 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:32.990523 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" podStartSLOduration=1.990506335 podStartE2EDuration="1.990506335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:32.989368382 +0000 UTC m=+988.548004880" watchObservedRunningTime="2026-11-03 14:28:32.990506335 +0000 UTC m=+988.549142822" Nov 03 14:28:38.236189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.236154 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm"] Nov 03 14:28:38.241673 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.241653 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.261651 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.261600 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm"] Nov 03 14:28:38.271273 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.271246 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-18b6d76b5772\"" Nov 03 14:28:38.271485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.271464 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-b28d34176fc3\"" Nov 03 14:28:38.271580 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.271564 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-e2e--29c79c46-dockercfg-xvjqk\"" Nov 03 14:28:38.271658 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.271625 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-95ac97e0ca1e\"" Nov 03 14:28:38.271738 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.271662 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-c88df00e610c\"" Nov 03 14:28:38.271795 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.271762 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"kube-root-ca.crt\"" Nov 03 14:28:38.273646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.273599 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"openshift-service-ca.crt\"" Nov 03 14:28:38.273766 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.273712 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b4361f\"" Nov 03 14:28:38.273766 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.273732 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"routing-overlay\"" Nov 03 14:28:38.369420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.369380 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46m2t\" (UniqueName: \"kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-kube-api-access-46m2t\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.369641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.369519 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.369755 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.369655 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.369846 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.369827 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.470864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.470824 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46m2t\" (UniqueName: \"kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-kube-api-access-46m2t\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.471044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.470896 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.471044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.470951 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.471188 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.471099 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.471775 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.471750 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.471839 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.471812 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.478184 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.474888 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.493545 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.493482 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46m2t\" (UniqueName: \"kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-kube-api-access-46m2t\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.555848 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.555812 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:38.863428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.863400 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm"] Nov 03 14:28:38.864806 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:28:38.864778 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2272b05_b105_4aed_bf4d_cb82c8b201d8.slice/crio-02678b9028dcfb7f98c0dc8bb6450e0833cdc10c61269dd5d0995f44393e8074 WatchSource:0}: Error finding container 02678b9028dcfb7f98c0dc8bb6450e0833cdc10c61269dd5d0995f44393e8074: Status 404 returned error can't find the container with id 02678b9028dcfb7f98c0dc8bb6450e0833cdc10c61269dd5d0995f44393e8074 Nov 03 14:28:38.961860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.961821 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" event={"ID":"f2272b05-b105-4aed-bf4d-cb82c8b201d8","Type":"ContainerStarted","Data":"57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f"} Nov 03 14:28:38.962020 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.961884 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" event={"ID":"f2272b05-b105-4aed-bf4d-cb82c8b201d8","Type":"ContainerStarted","Data":"02678b9028dcfb7f98c0dc8bb6450e0833cdc10c61269dd5d0995f44393e8074"} Nov 03 14:28:38.962020 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:38.962003 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:39.012329 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:39.012281 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" podStartSLOduration=1.012265929 podStartE2EDuration="1.012265929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:39.009817331 +0000 UTC m=+994.568453815" watchObservedRunningTime="2026-11-03 14:28:39.012265929 +0000 UTC m=+994.570902414" Nov 03 14:28:43.519952 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.519863 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s"] Nov 03 14:28:43.524242 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.524217 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.553106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.553000 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-maas-api-tls\") pod \"maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.553106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.553105 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfba3f9-8081-46b1-a80f-31734ffda48b-openshift-service-ca\") pod \"maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.553326 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.553140 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h6dk\" (UniqueName: \"kubernetes.io/projected/fdfba3f9-8081-46b1-a80f-31734ffda48b-kube-api-access-9h6dk\") pod \"maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.553408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.553381 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-metrics-certs\") pod \"maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.556215 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.556180 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cbd38938-serving-cert\"" Nov 03 14:28:43.560455 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.560430 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s"] Nov 03 14:28:43.654428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.654387 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-maas-api-tls\") pod \"maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.654594 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.654486 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfba3f9-8081-46b1-a80f-31734ffda48b-openshift-service-ca\") pod \"maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.654594 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.654519 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h6dk\" (UniqueName: \"kubernetes.io/projected/fdfba3f9-8081-46b1-a80f-31734ffda48b-kube-api-access-9h6dk\") pod \"maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.654772 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.654593 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-metrics-certs\") pod \"maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.655561 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.655535 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfba3f9-8081-46b1-a80f-31734ffda48b-openshift-service-ca\") pod \"maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.658062 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.658039 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-metrics-certs\") pod \"maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.663349 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.663320 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-maas-api-tls\") pod \"maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.680988 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.680964 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h6dk\" (UniqueName: \"kubernetes.io/projected/fdfba3f9-8081-46b1-a80f-31734ffda48b-kube-api-access-9h6dk\") pod \"maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.841654 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.841549 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:43.920823 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.919691 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" Nov 03 14:28:43.921269 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:43.921246 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" Nov 03 14:28:44.182428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:44.182397 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s"] Nov 03 14:28:45.098600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:45.090524 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" event={"ID":"fdfba3f9-8081-46b1-a80f-31734ffda48b","Type":"ContainerStarted","Data":"099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71"} Nov 03 14:28:45.098600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:45.092162 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" event={"ID":"fdfba3f9-8081-46b1-a80f-31734ffda48b","Type":"ContainerStarted","Data":"3b25324a7882b2b80e83d13cfecb72a08f4100788c800c8c563aeeb2d1157c62"} Nov 03 14:28:45.098600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:45.092381 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:45.139302 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:45.139255 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" podStartSLOduration=1.761198541 podStartE2EDuration="2.139240464s" totalImagesPullingTime="378.041923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:45.138542531 +0000 UTC m=+1000.697179015" watchObservedRunningTime="2026-11-03 14:28:45.139240464 +0000 UTC m=+1000.697876948" Nov 03 14:28:45.830072 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:45.830034 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm"] Nov 03 14:28:45.834429 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:45.834399 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:50.830657 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:50.830597 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" podUID="f2272b05-b105-4aed-bf4d-cb82c8b201d8" containerName="payload-processing" containerID="cri-o://57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f" gracePeriod=25 Nov 03 14:28:51.099366 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.099298 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:51.101770 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.101751 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:28:51.134990 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.134957 6389 generic.go:356] "Generic (PLEG): container finished" podID="f2272b05-b105-4aed-bf4d-cb82c8b201d8" containerID="57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f" exitCode=0 Nov 03 14:28:51.135137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.135000 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" Nov 03 14:28:51.135137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.135011 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" event={"ID":"f2272b05-b105-4aed-bf4d-cb82c8b201d8","Type":"ContainerDied","Data":"57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f"} Nov 03 14:28:51.135137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.135062 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm" event={"ID":"f2272b05-b105-4aed-bf4d-cb82c8b201d8","Type":"ContainerDied","Data":"02678b9028dcfb7f98c0dc8bb6450e0833cdc10c61269dd5d0995f44393e8074"} Nov 03 14:28:51.135137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.135108 6389 scope.go:122] "RemoveContainer" containerID="57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f" Nov 03 14:28:51.145722 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.145697 6389 scope.go:122] "RemoveContainer" containerID="57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f" Nov 03 14:28:51.146041 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:28:51.146018 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f\": container with ID starting with 57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f not found: ID does not exist" containerID="57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f" Nov 03 14:28:51.146123 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.146052 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f"} err="failed to get container status \"57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f\": rpc error: code = NotFound desc = could not find container \"57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f\": container with ID starting with 57c337a3fa0b4290aae9354cee75069ee816be2641895856d256ab84ae82dd2f not found: ID does not exist" Nov 03 14:28:51.221175 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.221133 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-provider-credentials\") pod \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " Nov 03 14:28:51.221357 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.221230 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-plugins-config-volume\") pod \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " Nov 03 14:28:51.221357 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.221273 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-kube-api-access-46m2t\" (UniqueName: \"kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-kube-api-access-46m2t\") pod \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " Nov 03 14:28:51.221475 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.221464 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-routing-overlay-volume\") pod \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\" (UID: \"f2272b05-b105-4aed-bf4d-cb82c8b201d8\") " Nov 03 14:28:51.222501 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.222474 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-plugins-config-volume" pod "f2272b05-b105-4aed-bf4d-cb82c8b201d8" (UID: "f2272b05-b105-4aed-bf4d-cb82c8b201d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:51.222843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.222799 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-routing-overlay-volume" pod "f2272b05-b105-4aed-bf4d-cb82c8b201d8" (UID: "f2272b05-b105-4aed-bf4d-cb82c8b201d8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:51.224801 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.224778 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:51.224928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.224805 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2272b05-b105-4aed-bf4d-cb82c8b201d8-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:51.226010 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.225973 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-kube-api-access-46m2t" pod "f2272b05-b105-4aed-bf4d-cb82c8b201d8" (UID: "f2272b05-b105-4aed-bf4d-cb82c8b201d8"). InnerVolumeSpecName "kube-api-access-46m2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:51.228295 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.228266 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-provider-credentials" pod "f2272b05-b105-4aed-bf4d-cb82c8b201d8" (UID: "f2272b05-b105-4aed-bf4d-cb82c8b201d8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:51.330214 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.330171 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:51.330214 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.330215 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46m2t\" (UniqueName: \"kubernetes.io/projected/f2272b05-b105-4aed-bf4d-cb82c8b201d8-kube-api-access-46m2t\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:28:51.488899 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.488864 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm"] Nov 03 14:28:51.492360 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:51.492329 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-c7fcxngsm"] Nov 03 14:28:52.618743 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:28:52.618714 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2272b05-b105-4aed-bf4d-cb82c8b201d8" path="/var/lib/kubelet/pods/f2272b05-b105-4aed-bf4d-cb82c8b201d8/volumes" Nov 03 14:28:58.375649 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:28:58.375596 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:29:04.124635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:04.085827 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:29:04.124635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:04.085907 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:29:04.711795 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:04.711750 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:29:04.712006 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:04.711831 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" podUID="30003229-866c-4a72-be45-37c3134e3159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:29:12.489584 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.489550 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d"] Nov 03 14:29:12.490396 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.490375 6389 cpu_manager.go:414] "containerMap: removing container" podUID="f2272b05-b105-4aed-bf4d-cb82c8b201d8" containerName="payload-processing" Nov 03 14:29:12.490515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.490398 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2272b05-b105-4aed-bf4d-cb82c8b201d8" containerName="payload-processing" Nov 03 14:29:12.490596 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.490583 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2272b05-b105-4aed-bf4d-cb82c8b201d8" containerName="payload-processing" Nov 03 14:29:12.496082 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.496041 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" Nov 03 14:29:12.532304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.517992 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d"] Nov 03 14:29:12.587661 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.568927 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4a67e25f\"" Nov 03 14:29:12.587661 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.569074 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4a67e25f-dockercfg-hsrd7\"" Nov 03 14:29:12.691452 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.691412 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dcec4aa-e7d6-4d20-b29b-bf7890788744-plugins-config-volume\") pod \"payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d\" (UID: \"2dcec4aa-e7d6-4d20-b29b-bf7890788744\") " pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" Nov 03 14:29:12.691671 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.691517 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8w2b\" (UniqueName: \"kubernetes.io/projected/2dcec4aa-e7d6-4d20-b29b-bf7890788744-kube-api-access-t8w2b\") pod \"payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d\" (UID: \"2dcec4aa-e7d6-4d20-b29b-bf7890788744\") " pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" Nov 03 14:29:12.792756 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.792662 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dcec4aa-e7d6-4d20-b29b-bf7890788744-plugins-config-volume\") pod \"payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d\" (UID: \"2dcec4aa-e7d6-4d20-b29b-bf7890788744\") " pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" Nov 03 14:29:12.792756 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.792743 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8w2b\" (UniqueName: \"kubernetes.io/projected/2dcec4aa-e7d6-4d20-b29b-bf7890788744-kube-api-access-t8w2b\") pod \"payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d\" (UID: \"2dcec4aa-e7d6-4d20-b29b-bf7890788744\") " pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" Nov 03 14:29:12.793421 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.793397 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dcec4aa-e7d6-4d20-b29b-bf7890788744-plugins-config-volume\") pod \"payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d\" (UID: \"2dcec4aa-e7d6-4d20-b29b-bf7890788744\") " pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" Nov 03 14:29:12.815471 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.815431 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8w2b\" (UniqueName: \"kubernetes.io/projected/2dcec4aa-e7d6-4d20-b29b-bf7890788744-kube-api-access-t8w2b\") pod \"payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d\" (UID: \"2dcec4aa-e7d6-4d20-b29b-bf7890788744\") " pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" Nov 03 14:29:12.981999 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:12.981960 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" Nov 03 14:29:13.296824 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:13.296761 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d"] Nov 03 14:29:13.298732 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:29:13.298703 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcec4aa_e7d6_4d20_b29b_bf7890788744.slice/crio-e82e18df22425da7bab6fa69be55e4a5b35ba1960dfa2e97fb2ea970f3d0a712 WatchSource:0}: Error finding container e82e18df22425da7bab6fa69be55e4a5b35ba1960dfa2e97fb2ea970f3d0a712: Status 404 returned error can't find the container with id e82e18df22425da7bab6fa69be55e4a5b35ba1960dfa2e97fb2ea970f3d0a712 Nov 03 14:29:13.417508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:13.417476 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" event={"ID":"2dcec4aa-e7d6-4d20-b29b-bf7890788744","Type":"ContainerStarted","Data":"4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a"} Nov 03 14:29:13.417693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:13.417524 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" event={"ID":"2dcec4aa-e7d6-4d20-b29b-bf7890788744","Type":"ContainerStarted","Data":"e82e18df22425da7bab6fa69be55e4a5b35ba1960dfa2e97fb2ea970f3d0a712"} Nov 03 14:29:13.419040 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:13.418792 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" Nov 03 14:29:13.463430 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:13.463363 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" podStartSLOduration=1.463343574 podStartE2EDuration="1.463343574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:13.461683163 +0000 UTC m=+1029.020319650" watchObservedRunningTime="2026-11-03 14:29:13.463343574 +0000 UTC m=+1029.021980061" Nov 03 14:29:14.894366 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:14.893995 6389 patch_prober.go:29] interesting pod/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 14:29:14.894366 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:14.894059 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 14:29:15.494735 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:15.494700 6389 generic.go:356] "Generic (PLEG): container finished" podID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerID="f583d8d261453361f0ae1e1e8c9b573ce24c4fa8733daf8f14b2acc5a494f627" exitCode=0 Nov 03 14:29:15.495918 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:15.495890 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" event={"ID":"5ba32398-976e-4f2d-9cf1-b00f04bb56ff","Type":"ContainerDied","Data":"f583d8d261453361f0ae1e1e8c9b573ce24c4fa8733daf8f14b2acc5a494f627"} Nov 03 14:29:15.496480 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:15.496458 6389 scope.go:122] "RemoveContainer" containerID="f583d8d261453361f0ae1e1e8c9b573ce24c4fa8733daf8f14b2acc5a494f627" Nov 03 14:29:15.508137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:15.505662 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:29:15.508137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:15.505754 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:29:15.508137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:15.505806 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:29:16.505061 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:16.505019 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" event={"ID":"5ba32398-976e-4f2d-9cf1-b00f04bb56ff","Type":"ContainerStarted","Data":"fb07f69e87f9f85b949cf86c3f7bc302917732a38d6bf9649d79186e963d2aaf"} Nov 03 14:29:16.917921 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:16.917893 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:29:17.917737 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:17.917702 6389 patch_prober.go:29] interesting pod/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 14:29:17.918128 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:17.917756 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 14:29:18.917403 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:18.917373 6389 patch_prober.go:29] interesting pod/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 14:29:18.917565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:18.917424 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 14:29:19.917532 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:19.917498 6389 patch_prober.go:29] interesting pod/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 14:29:19.917948 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:19.917549 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 14:29:20.917617 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:20.917580 6389 patch_prober.go:29] interesting pod/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 14:29:20.918015 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:20.917682 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 14:29:21.897022 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:21.896989 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc"] Nov 03 14:29:21.917533 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:21.917509 6389 patch_prober.go:29] interesting pod/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 14:29:21.917720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:21.917559 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 14:29:21.945840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:21.945805 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw"] Nov 03 14:29:23.040664 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:23.040595 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:29:23.041505 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:23.041482 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:29:23.042187 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:23.042160 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:29:25.498835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:25.498803 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" Nov 03 14:29:26.898739 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:26.898684 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" podUID="7b51c8bc-90ac-4257-8c51-dbcd40ac34bd" containerName="payload-processing" containerID="cri-o://b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e" gracePeriod=25 Nov 03 14:29:26.946370 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:26.946321 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" podUID="63a0d285-a6a2-455c-afb2-625e5a8dd0ef" containerName="payload-pre-processing" containerID="cri-o://31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14" gracePeriod=25 Nov 03 14:29:27.343764 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.343736 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" Nov 03 14:29:27.347838 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.347815 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" Nov 03 14:29:27.391623 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.390376 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-plugins-config-volume\") pod \"63a0d285-a6a2-455c-afb2-625e5a8dd0ef\" (UID: \"63a0d285-a6a2-455c-afb2-625e5a8dd0ef\") " Nov 03 14:29:27.391623 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.390442 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-kube-api-access-w7rpp\" (UniqueName: \"kubernetes.io/projected/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-kube-api-access-w7rpp\") pod \"63a0d285-a6a2-455c-afb2-625e5a8dd0ef\" (UID: \"63a0d285-a6a2-455c-afb2-625e5a8dd0ef\") " Nov 03 14:29:27.391623 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.390497 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-plugins-config-volume\") pod \"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd\" (UID: \"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd\") " Nov 03 14:29:27.391623 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.390737 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-kube-api-access-w7fpk\" (UniqueName: \"kubernetes.io/projected/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-kube-api-access-w7fpk\") pod \"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd\" (UID: \"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd\") " Nov 03 14:29:27.393595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.393567 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-plugins-config-volume" pod "63a0d285-a6a2-455c-afb2-625e5a8dd0ef" (UID: "63a0d285-a6a2-455c-afb2-625e5a8dd0ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:27.394197 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.394176 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-plugins-config-volume" pod "7b51c8bc-90ac-4257-8c51-dbcd40ac34bd" (UID: "7b51c8bc-90ac-4257-8c51-dbcd40ac34bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:27.414460 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.414377 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-kube-api-access-w7fpk" pod "7b51c8bc-90ac-4257-8c51-dbcd40ac34bd" (UID: "7b51c8bc-90ac-4257-8c51-dbcd40ac34bd"). InnerVolumeSpecName "kube-api-access-w7fpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:27.416755 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.416710 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-kube-api-access-w7rpp" pod "63a0d285-a6a2-455c-afb2-625e5a8dd0ef" (UID: "63a0d285-a6a2-455c-afb2-625e5a8dd0ef"). InnerVolumeSpecName "kube-api-access-w7rpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:27.495760 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.495667 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7fpk\" (UniqueName: \"kubernetes.io/projected/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-kube-api-access-w7fpk\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:27.495983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.495958 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:27.496057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.495986 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7rpp\" (UniqueName: \"kubernetes.io/projected/63a0d285-a6a2-455c-afb2-625e5a8dd0ef-kube-api-access-w7rpp\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:27.496057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.496001 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:27.612947 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.612909 6389 generic.go:356] "Generic (PLEG): container finished" podID="63a0d285-a6a2-455c-afb2-625e5a8dd0ef" containerID="31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14" exitCode=0 Nov 03 14:29:27.613149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.613023 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" event={"ID":"63a0d285-a6a2-455c-afb2-625e5a8dd0ef","Type":"ContainerDied","Data":"31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14"} Nov 03 14:29:27.613149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.613062 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" event={"ID":"63a0d285-a6a2-455c-afb2-625e5a8dd0ef","Type":"ContainerDied","Data":"98aa93b9cbf95d7029493d2dc1a19dec023e696f56a05488182cc4036b867589"} Nov 03 14:29:27.613149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.613089 6389 scope.go:122] "RemoveContainer" containerID="31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14" Nov 03 14:29:27.613327 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.613222 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw" Nov 03 14:29:27.617201 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.617170 6389 generic.go:356] "Generic (PLEG): container finished" podID="7b51c8bc-90ac-4257-8c51-dbcd40ac34bd" containerID="b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e" exitCode=0 Nov 03 14:29:27.617570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.617280 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" event={"ID":"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd","Type":"ContainerDied","Data":"b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e"} Nov 03 14:29:27.617570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.617319 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" event={"ID":"7b51c8bc-90ac-4257-8c51-dbcd40ac34bd","Type":"ContainerDied","Data":"d9ebc3187f8540e58a87828ee878aa8bd7540f984ac65aedc76aca6cd66de6db"} Nov 03 14:29:27.617570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.617391 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc" Nov 03 14:29:27.626739 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.626716 6389 scope.go:122] "RemoveContainer" containerID="31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14" Nov 03 14:29:27.627263 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:29:27.627234 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14\": container with ID starting with 31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14 not found: ID does not exist" containerID="31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14" Nov 03 14:29:27.627408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.627381 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14"} err="failed to get container status \"31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14\": rpc error: code = NotFound desc = could not find container \"31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14\": container with ID starting with 31f59c9b1c3b0e6629d1d27058b3cc78c4718636f5a548399cd0cf9ad7b79a14 not found: ID does not exist" Nov 03 14:29:27.627501 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.627484 6389 scope.go:122] "RemoveContainer" containerID="b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e" Nov 03 14:29:27.641757 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.640499 6389 scope.go:122] "RemoveContainer" containerID="b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e" Nov 03 14:29:27.641757 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:29:27.640877 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e\": container with ID starting with b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e not found: ID does not exist" containerID="b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e" Nov 03 14:29:27.641757 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.640912 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e"} err="failed to get container status \"b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e\": rpc error: code = NotFound desc = could not find container \"b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e\": container with ID starting with b9f0a83b5f7a8952b40f0df2887a7f0f4facc7b3a8fa45be212e21697d6a922e not found: ID does not exist" Nov 03 14:29:27.654644 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.654593 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb"] Nov 03 14:29:27.655928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.655900 6389 cpu_manager.go:414] "containerMap: removing container" podUID="63a0d285-a6a2-455c-afb2-625e5a8dd0ef" containerName="payload-pre-processing" Nov 03 14:29:27.655928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.655929 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a0d285-a6a2-455c-afb2-625e5a8dd0ef" containerName="payload-pre-processing" Nov 03 14:29:27.656112 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.655945 6389 cpu_manager.go:414] "containerMap: removing container" podUID="7b51c8bc-90ac-4257-8c51-dbcd40ac34bd" containerName="payload-processing" Nov 03 14:29:27.656112 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.655953 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b51c8bc-90ac-4257-8c51-dbcd40ac34bd" containerName="payload-processing" Nov 03 14:29:27.656112 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.656064 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a0d285-a6a2-455c-afb2-625e5a8dd0ef" containerName="payload-pre-processing" Nov 03 14:29:27.656112 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.656077 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b51c8bc-90ac-4257-8c51-dbcd40ac34bd" containerName="payload-processing" Nov 03 14:29:27.661379 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.661357 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:27.679646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.678429 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4a67e25f-serving-cert\"" Nov 03 14:29:27.705910 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.705131 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb"] Nov 03 14:29:27.705910 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.705207 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc"] Nov 03 14:29:27.706944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.706920 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-maas-api-tls\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:27.707074 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.706957 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9738db50-dced-4ca1-8ab0-305ac60d8e8d-openshift-service-ca\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:27.707074 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.707014 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-metrics-certs\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:27.707074 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.707036 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2prmh\" (UniqueName: \"kubernetes.io/projected/9738db50-dced-4ca1-8ab0-305ac60d8e8d-kube-api-access-2prmh\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:27.710319 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.710296 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbd38938-654b48cf79-mmknc"] Nov 03 14:29:27.735759 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.735725 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw"] Nov 03 14:29:27.743876 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.743842 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbd38938-5db46956c-mjgrw"] Nov 03 14:29:27.808443 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.808356 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-maas-api-tls\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:27.808443 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.808394 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9738db50-dced-4ca1-8ab0-305ac60d8e8d-openshift-service-ca\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:27.808443 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.808426 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-metrics-certs\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:27.808763 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.808448 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2prmh\" (UniqueName: \"kubernetes.io/projected/9738db50-dced-4ca1-8ab0-305ac60d8e8d-kube-api-access-2prmh\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:27.808763 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:29:27.808517 6389 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-serving-cert: secret "maas-api-e2e-mt-4a67e25f-serving-cert" not found Nov 03 14:29:27.808763 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:29:27.808586 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-maas-api-tls podName:9738db50-dced-4ca1-8ab0-305ac60d8e8d nodeName:}" failed. No retries permitted until 2026-11-03 14:29:28.308566661 +0000 UTC m=+1043.867203124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-maas-api-tls") pod "maas-api-e2e-mt-4a67e25f-57fd484774-768cb" (UID: "9738db50-dced-4ca1-8ab0-305ac60d8e8d") : secret "maas-api-e2e-mt-4a67e25f-serving-cert" not found Nov 03 14:29:27.809139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.809119 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9738db50-dced-4ca1-8ab0-305ac60d8e8d-openshift-service-ca\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:27.811128 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.811105 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-metrics-certs\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:27.833283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:27.833244 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2prmh\" (UniqueName: \"kubernetes.io/projected/9738db50-dced-4ca1-8ab0-305ac60d8e8d-kube-api-access-2prmh\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:28.326629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:28.326512 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-maas-api-tls\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:28.340688 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:28.335944 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-maas-api-tls\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-768cb\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:28.424704 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:29:28.423516 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:29:28.625729 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:28.624442 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:28.701116 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:28.677135 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a0d285-a6a2-455c-afb2-625e5a8dd0ef" path="/var/lib/kubelet/pods/63a0d285-a6a2-455c-afb2-625e5a8dd0ef/volumes" Nov 03 14:29:28.701116 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:28.677889 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b51c8bc-90ac-4257-8c51-dbcd40ac34bd" path="/var/lib/kubelet/pods/7b51c8bc-90ac-4257-8c51-dbcd40ac34bd/volumes" Nov 03 14:29:29.240187 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:29.240149 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb"] Nov 03 14:29:29.754687 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:29.741680 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" event={"ID":"9738db50-dced-4ca1-8ab0-305ac60d8e8d","Type":"ContainerStarted","Data":"103ec2b3a4101b67da33afc8c01c3d4b4d51576a75e961b6bd47ed5f0e5b483c"} Nov 03 14:29:30.799630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:30.793003 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" event={"ID":"9738db50-dced-4ca1-8ab0-305ac60d8e8d","Type":"ContainerStarted","Data":"cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d"} Nov 03 14:29:30.799630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:30.794055 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:37.814037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:37.814008 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:29:37.860033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:37.859974 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" podStartSLOduration=10.364648543 podStartE2EDuration="10.85995872s" totalImagesPullingTime="495.310177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:30.848958443 +0000 UTC m=+1046.407594940" watchObservedRunningTime="2026-11-03 14:29:37.85995872 +0000 UTC m=+1053.418595203" Nov 03 14:29:40.542859 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.542821 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s"] Nov 03 14:29:40.543377 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.543130 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" podUID="fdfba3f9-8081-46b1-a80f-31734ffda48b" containerName="maas-api" containerID="cri-o://099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71" gracePeriod=30 Nov 03 14:29:40.806899 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.806836 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:29:40.895025 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.894979 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-metrics-certs\") pod \"fdfba3f9-8081-46b1-a80f-31734ffda48b\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " Nov 03 14:29:40.900360 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.900105 6389 generic.go:356] "Generic (PLEG): container finished" podID="fdfba3f9-8081-46b1-a80f-31734ffda48b" containerID="099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71" exitCode=0 Nov 03 14:29:40.900524 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.900398 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-metrics-certs" pod "fdfba3f9-8081-46b1-a80f-31734ffda48b" (UID: "fdfba3f9-8081-46b1-a80f-31734ffda48b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:40.900524 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.900172 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" Nov 03 14:29:40.900725 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.900148 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" event={"ID":"fdfba3f9-8081-46b1-a80f-31734ffda48b","Type":"ContainerDied","Data":"099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71"} Nov 03 14:29:40.900725 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.900599 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s" event={"ID":"fdfba3f9-8081-46b1-a80f-31734ffda48b","Type":"ContainerDied","Data":"3b25324a7882b2b80e83d13cfecb72a08f4100788c800c8c563aeeb2d1157c62"} Nov 03 14:29:40.900725 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.900670 6389 scope.go:122] "RemoveContainer" containerID="099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71" Nov 03 14:29:40.900966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.900946 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-maas-api-tls\") pod \"fdfba3f9-8081-46b1-a80f-31734ffda48b\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " Nov 03 14:29:40.901875 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.901840 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfba3f9-8081-46b1-a80f-31734ffda48b-kube-api-access-9h6dk\" (UniqueName: \"kubernetes.io/projected/fdfba3f9-8081-46b1-a80f-31734ffda48b-kube-api-access-9h6dk\") pod \"fdfba3f9-8081-46b1-a80f-31734ffda48b\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " Nov 03 14:29:40.902136 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.902117 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdfba3f9-8081-46b1-a80f-31734ffda48b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfba3f9-8081-46b1-a80f-31734ffda48b-openshift-service-ca\") pod \"fdfba3f9-8081-46b1-a80f-31734ffda48b\" (UID: \"fdfba3f9-8081-46b1-a80f-31734ffda48b\") " Nov 03 14:29:40.902854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.902576 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfba3f9-8081-46b1-a80f-31734ffda48b-openshift-service-ca" pod "fdfba3f9-8081-46b1-a80f-31734ffda48b" (UID: "fdfba3f9-8081-46b1-a80f-31734ffda48b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:40.902854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.902812 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfba3f9-8081-46b1-a80f-31734ffda48b-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:40.902854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.902830 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:40.905893 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.905860 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfba3f9-8081-46b1-a80f-31734ffda48b-kube-api-access-9h6dk" pod "fdfba3f9-8081-46b1-a80f-31734ffda48b" (UID: "fdfba3f9-8081-46b1-a80f-31734ffda48b"). InnerVolumeSpecName "kube-api-access-9h6dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:40.908829 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.908804 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-maas-api-tls" pod "fdfba3f9-8081-46b1-a80f-31734ffda48b" (UID: "fdfba3f9-8081-46b1-a80f-31734ffda48b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:40.943226 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.943113 6389 scope.go:122] "RemoveContainer" containerID="099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71" Nov 03 14:29:40.943561 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:29:40.943535 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71\": container with ID starting with 099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71 not found: ID does not exist" containerID="099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71" Nov 03 14:29:40.943702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:40.943575 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71"} err="failed to get container status \"099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71\": rpc error: code = NotFound desc = could not find container \"099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71\": container with ID starting with 099dfcb94a422fa1207598393b8b2f61ecee1cb6cd517ce558c8607ec3dadc71 not found: ID does not exist" Nov 03 14:29:41.004033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:41.004004 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h6dk\" (UniqueName: \"kubernetes.io/projected/fdfba3f9-8081-46b1-a80f-31734ffda48b-kube-api-access-9h6dk\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:41.004033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:41.004032 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfba3f9-8081-46b1-a80f-31734ffda48b-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:41.250362 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:41.250325 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s"] Nov 03 14:29:41.262751 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:41.262725 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbd38938-5b5bfc6fbf-jzm6s"] Nov 03 14:29:41.655004 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:41.654907 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk"] Nov 03 14:29:41.655597 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:41.655119 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" containerID="cri-o://fb07f69e87f9f85b949cf86c3f7bc302917732a38d6bf9649d79186e963d2aaf" gracePeriod=30 Nov 03 14:29:42.638685 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:42.637583 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdfba3f9-8081-46b1-a80f-31734ffda48b" path="/var/lib/kubelet/pods/fdfba3f9-8081-46b1-a80f-31734ffda48b/volumes" Nov 03 14:29:45.505894 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.505863 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg"] Nov 03 14:29:45.506546 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.506530 6389 cpu_manager.go:414] "containerMap: removing container" podUID="fdfba3f9-8081-46b1-a80f-31734ffda48b" containerName="maas-api" Nov 03 14:29:45.506603 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.506548 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdfba3f9-8081-46b1-a80f-31734ffda48b" containerName="maas-api" Nov 03 14:29:45.506682 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.506669 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdfba3f9-8081-46b1-a80f-31734ffda48b" containerName="maas-api" Nov 03 14:29:45.511692 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.511666 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.537334 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.537289 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg"] Nov 03 14:29:45.587440 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.587387 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.587647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.587516 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.587714 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.587649 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.587714 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.587681 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.587826 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.587772 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86h9w\" (UniqueName: \"kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-kube-api-access-86h9w\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.587882 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.587837 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.587967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.587952 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.588023 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.587988 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.588157 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.588133 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.689056 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689021 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.689265 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689074 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.689265 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689118 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.689265 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689185 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.689265 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689202 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.689265 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689225 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86h9w\" (UniqueName: \"kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-kube-api-access-86h9w\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.689554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689316 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.689554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689468 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.689554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689506 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.689751 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689647 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.689751 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689686 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.689751 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689736 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.690106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.689753 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.690258 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.690238 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.691836 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.691807 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.691950 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.691887 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.711050 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.711018 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.711235 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.711214 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86h9w\" (UniqueName: \"kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-kube-api-access-86h9w\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:45.825034 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:45.824959 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:46.143263 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:46.143238 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg"] Nov 03 14:29:46.144283 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:29:46.144259 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103e79b4_ac7a_4f53_affe_8a06ef5a3d46.slice/crio-9ef45ad5203cc3202775a79fc809293fb827f84c1fefe48f95c7268060968f4c WatchSource:0}: Error finding container 9ef45ad5203cc3202775a79fc809293fb827f84c1fefe48f95c7268060968f4c: Status 404 returned error can't find the container with id 9ef45ad5203cc3202775a79fc809293fb827f84c1fefe48f95c7268060968f4c Nov 03 14:29:46.146778 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:46.146749 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:29:46.146860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:46.146813 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:29:46.146860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:46.146840 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:29:46.958853 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:46.958826 6389 generic.go:356] "Generic (PLEG): container finished" podID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerID="fb07f69e87f9f85b949cf86c3f7bc302917732a38d6bf9649d79186e963d2aaf" exitCode=0 Nov 03 14:29:46.959340 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:46.958866 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" event={"ID":"5ba32398-976e-4f2d-9cf1-b00f04bb56ff","Type":"ContainerDied","Data":"fb07f69e87f9f85b949cf86c3f7bc302917732a38d6bf9649d79186e963d2aaf"} Nov 03 14:29:46.959340 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:46.958929 6389 scope.go:122] "RemoveContainer" containerID="f583d8d261453361f0ae1e1e8c9b573ce24c4fa8733daf8f14b2acc5a494f627" Nov 03 14:29:46.961082 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:46.961058 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" event={"ID":"103e79b4-ac7a-4f53-affe-8a06ef5a3d46","Type":"ContainerStarted","Data":"83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c"} Nov 03 14:29:46.961227 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:46.961103 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" event={"ID":"103e79b4-ac7a-4f53-affe-8a06ef5a3d46","Type":"ContainerStarted","Data":"9ef45ad5203cc3202775a79fc809293fb827f84c1fefe48f95c7268060968f4c"} Nov 03 14:29:46.986000 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:46.985982 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:29:47.046661 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.046586 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" podStartSLOduration=2.046570618 podStartE2EDuration="2.046570618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:47.028155417 +0000 UTC m=+1062.586791901" watchObservedRunningTime="2026-11-03 14:29:47.046570618 +0000 UTC m=+1062.605207102" Nov 03 14:29:47.108982 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.108952 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-kube-api-access-h4972\" (UniqueName: \"kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-kube-api-access-h4972\") pod \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " Nov 03 14:29:47.109184 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.108998 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-podinfo\") pod \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " Nov 03 14:29:47.109184 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.109031 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-certs\") pod \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " Nov 03 14:29:47.109184 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.109106 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-envoy\") pod \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " Nov 03 14:29:47.109184 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.109182 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-token\" (UniqueName: \"kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-token\") pod \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " Nov 03 14:29:47.109391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.109231 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-credential-socket\") pod \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " Nov 03 14:29:47.109391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.109281 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-data\") pod \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " Nov 03 14:29:47.109485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.109443 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-socket\") pod \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " Nov 03 14:29:47.109535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.109481 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istiod-ca-cert\") pod \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\" (UID: \"5ba32398-976e-4f2d-9cf1-b00f04bb56ff\") " Nov 03 14:29:47.111036 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.110903 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-socket" pod "5ba32398-976e-4f2d-9cf1-b00f04bb56ff" (UID: "5ba32398-976e-4f2d-9cf1-b00f04bb56ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:47.111036 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.110918 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-credential-socket" pod "5ba32398-976e-4f2d-9cf1-b00f04bb56ff" (UID: "5ba32398-976e-4f2d-9cf1-b00f04bb56ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:47.111246 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.111115 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-certs" pod "5ba32398-976e-4f2d-9cf1-b00f04bb56ff" (UID: "5ba32398-976e-4f2d-9cf1-b00f04bb56ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:47.111246 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.111173 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-data" pod "5ba32398-976e-4f2d-9cf1-b00f04bb56ff" (UID: "5ba32398-976e-4f2d-9cf1-b00f04bb56ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:47.111362 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.111348 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istiod-ca-cert" pod "5ba32398-976e-4f2d-9cf1-b00f04bb56ff" (UID: "5ba32398-976e-4f2d-9cf1-b00f04bb56ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:47.112269 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.112226 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-envoy" pod "5ba32398-976e-4f2d-9cf1-b00f04bb56ff" (UID: "5ba32398-976e-4f2d-9cf1-b00f04bb56ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:47.112269 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.112249 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-token" pod "5ba32398-976e-4f2d-9cf1-b00f04bb56ff" (UID: "5ba32398-976e-4f2d-9cf1-b00f04bb56ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:47.113013 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.112930 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-kube-api-access-h4972" pod "5ba32398-976e-4f2d-9cf1-b00f04bb56ff" (UID: "5ba32398-976e-4f2d-9cf1-b00f04bb56ff"). InnerVolumeSpecName "kube-api-access-h4972". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:47.114259 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.114230 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-podinfo" pod "5ba32398-976e-4f2d-9cf1-b00f04bb56ff" (UID: "5ba32398-976e-4f2d-9cf1-b00f04bb56ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:29:47.211961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.211875 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:47.211961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.211905 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:47.211961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.211914 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:47.211961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.211925 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:47.211961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.211933 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4972\" (UniqueName: \"kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-kube-api-access-h4972\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:47.211961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.211941 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:47.211961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.211951 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:47.211961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.211959 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:47.211961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.211966 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba32398-976e-4f2d-9cf1-b00f04bb56ff-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:29:47.585807 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.585720 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc"] Nov 03 14:29:47.586368 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.586349 6389 cpu_manager.go:414] "containerMap: removing container" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" Nov 03 14:29:47.586473 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.586371 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" Nov 03 14:29:47.586473 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.586412 6389 cpu_manager.go:414] "containerMap: removing container" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" Nov 03 14:29:47.586473 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.586420 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" Nov 03 14:29:47.586635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.586552 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" Nov 03 14:29:47.586635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.586570 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" containerName="istio-proxy" Nov 03 14:29:47.591291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.591269 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" Nov 03 14:29:47.613411 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.613383 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d608daf1\"" Nov 03 14:29:47.616035 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.616004 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d608daf1-dockercfg-z4z6q\"" Nov 03 14:29:47.715630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.709396 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc"] Nov 03 14:29:47.735635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.722433 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rn6\" (UniqueName: \"kubernetes.io/projected/ecb01978-2d07-4701-bfa6-81a0eeced7d3-kube-api-access-27rn6\") pod \"payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc\" (UID: \"ecb01978-2d07-4701-bfa6-81a0eeced7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" Nov 03 14:29:47.759634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.736483 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb01978-2d07-4701-bfa6-81a0eeced7d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc\" (UID: \"ecb01978-2d07-4701-bfa6-81a0eeced7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" Nov 03 14:29:47.830629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.830528 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:47.895757 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.894774 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 14:29:47.895757 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.894843 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" podUID="103e79b4-ac7a-4f53-affe-8a06ef5a3d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 14:29:47.897666 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.896413 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb01978-2d07-4701-bfa6-81a0eeced7d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc\" (UID: \"ecb01978-2d07-4701-bfa6-81a0eeced7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" Nov 03 14:29:47.897666 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.896595 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rn6\" (UniqueName: \"kubernetes.io/projected/ecb01978-2d07-4701-bfa6-81a0eeced7d3-kube-api-access-27rn6\") pod \"payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc\" (UID: \"ecb01978-2d07-4701-bfa6-81a0eeced7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" Nov 03 14:29:47.900275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.898036 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb01978-2d07-4701-bfa6-81a0eeced7d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc\" (UID: \"ecb01978-2d07-4701-bfa6-81a0eeced7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" Nov 03 14:29:47.900275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.898095 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh"] Nov 03 14:29:47.907925 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.907904 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh"] Nov 03 14:29:47.908031 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.907994 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" Nov 03 14:29:47.949685 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.919139 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rn6\" (UniqueName: \"kubernetes.io/projected/ecb01978-2d07-4701-bfa6-81a0eeced7d3-kube-api-access-27rn6\") pod \"payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc\" (UID: \"ecb01978-2d07-4701-bfa6-81a0eeced7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" Nov 03 14:29:47.949685 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.936841 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" Nov 03 14:29:47.970491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.970468 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" Nov 03 14:29:47.970901 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.970465 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk" event={"ID":"5ba32398-976e-4f2d-9cf1-b00f04bb56ff","Type":"ContainerDied","Data":"e6e8df9f336ccd667fab6f46df2d6edd78f7d90952f4daa38576a4316fc9f1f0"} Nov 03 14:29:47.970901 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:47.970632 6389 scope.go:122] "RemoveContainer" containerID="fb07f69e87f9f85b949cf86c3f7bc302917732a38d6bf9649d79186e963d2aaf" Nov 03 14:29:48.084060 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.084030 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk"] Nov 03 14:29:48.089293 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.089269 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cbd38938-openshift-default-55744f6757-mdddk"] Nov 03 14:29:48.099314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.099286 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09cd02dc-05d6-47f2-82e3-b5399e421025-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh\" (UID: \"09cd02dc-05d6-47f2-82e3-b5399e421025\") " pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" Nov 03 14:29:48.099469 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.099334 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdtv6\" (UniqueName: \"kubernetes.io/projected/09cd02dc-05d6-47f2-82e3-b5399e421025-kube-api-access-hdtv6\") pod \"payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh\" (UID: \"09cd02dc-05d6-47f2-82e3-b5399e421025\") " pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" Nov 03 14:29:48.200405 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.200364 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09cd02dc-05d6-47f2-82e3-b5399e421025-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh\" (UID: \"09cd02dc-05d6-47f2-82e3-b5399e421025\") " pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" Nov 03 14:29:48.200405 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.200408 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdtv6\" (UniqueName: \"kubernetes.io/projected/09cd02dc-05d6-47f2-82e3-b5399e421025-kube-api-access-hdtv6\") pod \"payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh\" (UID: \"09cd02dc-05d6-47f2-82e3-b5399e421025\") " pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" Nov 03 14:29:48.201186 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.201166 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09cd02dc-05d6-47f2-82e3-b5399e421025-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh\" (UID: \"09cd02dc-05d6-47f2-82e3-b5399e421025\") " pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" Nov 03 14:29:48.248551 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.248516 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdtv6\" (UniqueName: \"kubernetes.io/projected/09cd02dc-05d6-47f2-82e3-b5399e421025-kube-api-access-hdtv6\") pod \"payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh\" (UID: \"09cd02dc-05d6-47f2-82e3-b5399e421025\") " pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" Nov 03 14:29:48.265690 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:29:48.265662 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb01978_2d07_4701_bfa6_81a0eeced7d3.slice/crio-8297d4dcdd27318a76b8f0ac9bf043005cd690ddfa9e53a9b37a4e15686da134 WatchSource:0}: Error finding container 8297d4dcdd27318a76b8f0ac9bf043005cd690ddfa9e53a9b37a4e15686da134: Status 404 returned error can't find the container with id 8297d4dcdd27318a76b8f0ac9bf043005cd690ddfa9e53a9b37a4e15686da134 Nov 03 14:29:48.266330 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.266309 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc"] Nov 03 14:29:48.545728 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.544847 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" Nov 03 14:29:48.631278 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.630744 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba32398-976e-4f2d-9cf1-b00f04bb56ff" path="/var/lib/kubelet/pods/5ba32398-976e-4f2d-9cf1-b00f04bb56ff/volumes" Nov 03 14:29:48.843861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.842102 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 14:29:48.843861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.842175 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" podUID="103e79b4-ac7a-4f53-affe-8a06ef5a3d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 14:29:48.926633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:48.925893 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh"] Nov 03 14:29:49.012683 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:49.012655 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" event={"ID":"ecb01978-2d07-4701-bfa6-81a0eeced7d3","Type":"ContainerStarted","Data":"04ed5dedb8d97478ad7b5f6bfb67d3cdbf23cf35be9ec914ea4a699a58288483"} Nov 03 14:29:49.013079 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:49.012716 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" event={"ID":"ecb01978-2d07-4701-bfa6-81a0eeced7d3","Type":"ContainerStarted","Data":"8297d4dcdd27318a76b8f0ac9bf043005cd690ddfa9e53a9b37a4e15686da134"} Nov 03 14:29:49.013079 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:49.012768 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" Nov 03 14:29:49.061729 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:49.059514 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" podStartSLOduration=2.059494497 podStartE2EDuration="2.059494497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:49.05934871 +0000 UTC m=+1064.617985194" watchObservedRunningTime="2026-11-03 14:29:49.059494497 +0000 UTC m=+1064.618130982" Nov 03 14:29:49.828635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:49.827091 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 14:29:49.828635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:49.827193 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" podUID="103e79b4-ac7a-4f53-affe-8a06ef5a3d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 14:29:50.048289 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:50.048254 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" event={"ID":"09cd02dc-05d6-47f2-82e3-b5399e421025","Type":"ContainerStarted","Data":"c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d"} Nov 03 14:29:50.048731 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:50.048314 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" event={"ID":"09cd02dc-05d6-47f2-82e3-b5399e421025","Type":"ContainerStarted","Data":"e11b122856d60d7728de01f427d03f3b6c325b381e4fec7e19744213256ab22a"} Nov 03 14:29:50.048731 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:50.048635 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" Nov 03 14:29:50.096508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:50.096406 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" podStartSLOduration=3.096392296 podStartE2EDuration="3.096392296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:50.094480253 +0000 UTC m=+1065.653116736" watchObservedRunningTime="2026-11-03 14:29:50.096392296 +0000 UTC m=+1065.655028780" Nov 03 14:29:50.839634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:50.831212 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 14:29:50.839634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:50.831271 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" podUID="103e79b4-ac7a-4f53-affe-8a06ef5a3d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 14:29:51.828046 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:51.828009 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 14:29:51.828692 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:51.828658 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" podUID="103e79b4-ac7a-4f53-affe-8a06ef5a3d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 14:29:53.868775 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:53.868732 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:29:53.869507 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:53.869471 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" podUID="103e79b4-ac7a-4f53-affe-8a06ef5a3d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:29:54.400025 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:54.399991 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:54.401771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:54.401748 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:54.402828 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:29:54.402807 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:29:58.375804 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:29:58.375767 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:30:00.052207 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:00.052172 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" Nov 03 14:30:01.099892 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:01.099862 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" Nov 03 14:30:03.857084 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:03.857049 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct"] Nov 03 14:30:03.862328 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:03.862310 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:03.879697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:03.879679 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d608daf1-serving-cert\"" Nov 03 14:30:03.895646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:03.895586 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct"] Nov 03 14:30:03.976762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:03.976730 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-metrics-certs\") pod \"maas-api-e2e-migrate-d608daf1-64d4888778-7tvct\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:03.976924 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:03.976906 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4flmh\" (UniqueName: \"kubernetes.io/projected/68678292-3210-477f-95bc-3181f870950e-kube-api-access-4flmh\") pod \"maas-api-e2e-migrate-d608daf1-64d4888778-7tvct\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:03.976966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:03.976954 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68678292-3210-477f-95bc-3181f870950e-openshift-service-ca\") pod \"maas-api-e2e-migrate-d608daf1-64d4888778-7tvct\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:03.977093 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:03.977080 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-maas-api-tls\") pod \"maas-api-e2e-migrate-d608daf1-64d4888778-7tvct\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:04.078728 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:04.078698 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4flmh\" (UniqueName: \"kubernetes.io/projected/68678292-3210-477f-95bc-3181f870950e-kube-api-access-4flmh\") pod \"maas-api-e2e-migrate-d608daf1-64d4888778-7tvct\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:04.078908 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:04.078747 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68678292-3210-477f-95bc-3181f870950e-openshift-service-ca\") pod \"maas-api-e2e-migrate-d608daf1-64d4888778-7tvct\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:04.078908 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:04.078823 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-maas-api-tls\") pod \"maas-api-e2e-migrate-d608daf1-64d4888778-7tvct\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:04.078908 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:04.078884 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-metrics-certs\") pod \"maas-api-e2e-migrate-d608daf1-64d4888778-7tvct\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:04.079661 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:04.079542 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68678292-3210-477f-95bc-3181f870950e-openshift-service-ca\") pod \"maas-api-e2e-migrate-d608daf1-64d4888778-7tvct\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:04.081469 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:04.081451 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-metrics-certs\") pod \"maas-api-e2e-migrate-d608daf1-64d4888778-7tvct\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:04.081577 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:04.081489 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-maas-api-tls\") pod \"maas-api-e2e-migrate-d608daf1-64d4888778-7tvct\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:04.143746 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:04.141321 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4flmh\" (UniqueName: \"kubernetes.io/projected/68678292-3210-477f-95bc-3181f870950e-kube-api-access-4flmh\") pod \"maas-api-e2e-migrate-d608daf1-64d4888778-7tvct\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:04.176112 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:04.175836 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:04.493346 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:04.493319 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct"] Nov 03 14:30:04.494988 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:30:04.494964 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68678292_3210_477f_95bc_3181f870950e.slice/crio-e4b887287b9331490f769ef5fcecf509550a3b6ebdb2d63199803d9ab1bd3e44 WatchSource:0}: Error finding container e4b887287b9331490f769ef5fcecf509550a3b6ebdb2d63199803d9ab1bd3e44: Status 404 returned error can't find the container with id e4b887287b9331490f769ef5fcecf509550a3b6ebdb2d63199803d9ab1bd3e44 Nov 03 14:30:05.269309 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:05.269269 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" event={"ID":"68678292-3210-477f-95bc-3181f870950e","Type":"ContainerStarted","Data":"4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd"} Nov 03 14:30:05.269780 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:05.269480 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:05.269780 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:05.269519 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" event={"ID":"68678292-3210-477f-95bc-3181f870950e","Type":"ContainerStarted","Data":"e4b887287b9331490f769ef5fcecf509550a3b6ebdb2d63199803d9ab1bd3e44"} Nov 03 14:30:05.313894 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:05.313841 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" podStartSLOduration=1.899236781 podStartE2EDuration="2.313826405s" totalImagesPullingTime="414.589624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:05.312069943 +0000 UTC m=+1080.870706431" watchObservedRunningTime="2026-11-03 14:30:05.313826405 +0000 UTC m=+1080.872462889" Nov 03 14:30:11.282736 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:11.282706 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:17.527527 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.527492 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5"] Nov 03 14:30:17.532924 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.532906 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.566801 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.566770 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5"] Nov 03 14:30:17.628742 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.628711 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.628742 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.628747 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.628987 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.628815 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.629056 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.628999 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.629056 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.629046 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.629164 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.629086 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.629218 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.629196 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.629292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.629276 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkvb5\" (UniqueName: \"kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-kube-api-access-wkvb5\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.629355 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.629318 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.730707 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.730675 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.730902 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.730712 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.730902 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.730843 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.731026 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.730913 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvb5\" (UniqueName: \"kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-kube-api-access-wkvb5\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.731026 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.730962 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.731188 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.731149 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.731333 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.731311 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.731424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.731341 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.731424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.731399 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.731759 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.731473 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.731759 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.731532 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.731880 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.731782 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.731943 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.731877 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.732321 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.732296 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.733318 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.733296 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.733578 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.733557 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.753632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.753578 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.753819 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.753799 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvb5\" (UniqueName: \"kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-kube-api-access-wkvb5\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:17.850396 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:17.850292 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:18.183784 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:18.183755 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5"] Nov 03 14:30:18.184977 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:30:18.184950 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8858f0b_63bc_4fc7_9626_fac6e74a9352.slice/crio-6155ef2b0c948fda3eb70400a81c110093f37b97f07c536bac5881c1460388eb WatchSource:0}: Error finding container 6155ef2b0c948fda3eb70400a81c110093f37b97f07c536bac5881c1460388eb: Status 404 returned error can't find the container with id 6155ef2b0c948fda3eb70400a81c110093f37b97f07c536bac5881c1460388eb Nov 03 14:30:18.187662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:18.187629 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:30:18.187768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:18.187703 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:30:18.187768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:18.187748 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:30:18.376633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:18.375985 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" event={"ID":"b8858f0b-63bc-4fc7-9626-fac6e74a9352","Type":"ContainerStarted","Data":"5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0"} Nov 03 14:30:18.376633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:18.376052 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" event={"ID":"b8858f0b-63bc-4fc7-9626-fac6e74a9352","Type":"ContainerStarted","Data":"6155ef2b0c948fda3eb70400a81c110093f37b97f07c536bac5881c1460388eb"} Nov 03 14:30:18.436365 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:18.436262 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" podStartSLOduration=1.436246952 podStartE2EDuration="1.436246952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:18.434136945 +0000 UTC m=+1093.992773431" watchObservedRunningTime="2026-11-03 14:30:18.436246952 +0000 UTC m=+1093.994883435" Nov 03 14:30:18.850759 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:18.850683 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:19.852515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:19.852472 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 14:30:19.853076 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:19.852542 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" podUID="b8858f0b-63bc-4fc7-9626-fac6e74a9352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 14:30:20.851495 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:20.851006 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 14:30:20.851495 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:20.851068 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" podUID="b8858f0b-63bc-4fc7-9626-fac6e74a9352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 14:30:21.855657 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:21.853076 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 14:30:21.855657 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:21.853140 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" podUID="b8858f0b-63bc-4fc7-9626-fac6e74a9352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 14:30:22.911309 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:22.911272 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 14:30:22.941250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:22.929151 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" podUID="b8858f0b-63bc-4fc7-9626-fac6e74a9352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 14:30:23.851057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:23.851017 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 14:30:23.851446 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:23.851412 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" podUID="b8858f0b-63bc-4fc7-9626-fac6e74a9352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 14:30:24.851407 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:24.851365 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 14:30:24.852028 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:24.851444 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" podUID="b8858f0b-63bc-4fc7-9626-fac6e74a9352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 14:30:26.106275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:26.106244 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:26.106870 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:26.106645 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:26.107765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:26.107740 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:30:28.373941 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:30:28.373905 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:30:37.395689 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:37.395653 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh"] Nov 03 14:30:37.429947 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:37.429919 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc"] Nov 03 14:30:42.285288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.285254 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct"] Nov 03 14:30:42.285874 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.285511 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" podUID="68678292-3210-477f-95bc-3181f870950e" containerName="maas-api" containerID="cri-o://4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd" gracePeriod=30 Nov 03 14:30:42.397033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.396986 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" podUID="09cd02dc-05d6-47f2-82e3-b5399e421025" containerName="payload-processing" containerID="cri-o://c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d" gracePeriod=25 Nov 03 14:30:42.430231 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.430189 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" podUID="ecb01978-2d07-4701-bfa6-81a0eeced7d3" containerName="payload-pre-processing" containerID="cri-o://04ed5dedb8d97478ad7b5f6bfb67d3cdbf23cf35be9ec914ea4a699a58288483" gracePeriod=25 Nov 03 14:30:42.728428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.728400 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:42.857995 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.857972 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" Nov 03 14:30:42.893055 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.893023 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68678292-3210-477f-95bc-3181f870950e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68678292-3210-477f-95bc-3181f870950e-openshift-service-ca\") pod \"68678292-3210-477f-95bc-3181f870950e\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " Nov 03 14:30:42.893282 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.893099 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68678292-3210-477f-95bc-3181f870950e-kube-api-access-4flmh\" (UniqueName: \"kubernetes.io/projected/68678292-3210-477f-95bc-3181f870950e-kube-api-access-4flmh\") pod \"68678292-3210-477f-95bc-3181f870950e\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " Nov 03 14:30:42.893282 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.893176 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-maas-api-tls\") pod \"68678292-3210-477f-95bc-3181f870950e\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " Nov 03 14:30:42.893282 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.893268 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-metrics-certs\") pod \"68678292-3210-477f-95bc-3181f870950e\" (UID: \"68678292-3210-477f-95bc-3181f870950e\") " Nov 03 14:30:42.893927 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.893875 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68678292-3210-477f-95bc-3181f870950e-openshift-service-ca" pod "68678292-3210-477f-95bc-3181f870950e" (UID: "68678292-3210-477f-95bc-3181f870950e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:42.895926 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.895906 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-metrics-certs" pod "68678292-3210-477f-95bc-3181f870950e" (UID: "68678292-3210-477f-95bc-3181f870950e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:42.911791 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.907747 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-maas-api-tls" pod "68678292-3210-477f-95bc-3181f870950e" (UID: "68678292-3210-477f-95bc-3181f870950e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:42.911791 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.907784 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68678292-3210-477f-95bc-3181f870950e-kube-api-access-4flmh" pod "68678292-3210-477f-95bc-3181f870950e" (UID: "68678292-3210-477f-95bc-3181f870950e"). InnerVolumeSpecName "kube-api-access-4flmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:42.937663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.931818 6389 generic.go:356] "Generic (PLEG): container finished" podID="68678292-3210-477f-95bc-3181f870950e" containerID="4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd" exitCode=0 Nov 03 14:30:42.937663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.931985 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" event={"ID":"68678292-3210-477f-95bc-3181f870950e","Type":"ContainerDied","Data":"4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd"} Nov 03 14:30:42.937663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.932042 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" event={"ID":"68678292-3210-477f-95bc-3181f870950e","Type":"ContainerDied","Data":"e4b887287b9331490f769ef5fcecf509550a3b6ebdb2d63199803d9ab1bd3e44"} Nov 03 14:30:42.937663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.932066 6389 scope.go:122] "RemoveContainer" containerID="4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd" Nov 03 14:30:42.937663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.932177 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct" Nov 03 14:30:42.937663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.936694 6389 generic.go:356] "Generic (PLEG): container finished" podID="ecb01978-2d07-4701-bfa6-81a0eeced7d3" containerID="04ed5dedb8d97478ad7b5f6bfb67d3cdbf23cf35be9ec914ea4a699a58288483" exitCode=0 Nov 03 14:30:42.937663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.936781 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" event={"ID":"ecb01978-2d07-4701-bfa6-81a0eeced7d3","Type":"ContainerDied","Data":"04ed5dedb8d97478ad7b5f6bfb67d3cdbf23cf35be9ec914ea4a699a58288483"} Nov 03 14:30:42.950848 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.948160 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" Nov 03 14:30:42.952649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.952502 6389 generic.go:356] "Generic (PLEG): container finished" podID="09cd02dc-05d6-47f2-82e3-b5399e421025" containerID="c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d" exitCode=0 Nov 03 14:30:42.952649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.952565 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" Nov 03 14:30:42.952649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.952586 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" event={"ID":"09cd02dc-05d6-47f2-82e3-b5399e421025","Type":"ContainerDied","Data":"c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d"} Nov 03 14:30:42.952918 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.952668 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh" event={"ID":"09cd02dc-05d6-47f2-82e3-b5399e421025","Type":"ContainerDied","Data":"e11b122856d60d7728de01f427d03f3b6c325b381e4fec7e19744213256ab22a"} Nov 03 14:30:42.955406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.955390 6389 scope.go:122] "RemoveContainer" containerID="4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd" Nov 03 14:30:42.955789 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:30:42.955763 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd\": container with ID starting with 4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd not found: ID does not exist" containerID="4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd" Nov 03 14:30:42.955878 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.955801 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd"} err="failed to get container status \"4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd\": rpc error: code = NotFound desc = could not find container \"4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd\": container with ID starting with 4e52dafd2916d51748b0ab14aa871e434a2d5eecdc46f17be5448e7d815c58fd not found: ID does not exist" Nov 03 14:30:42.955878 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.955828 6389 scope.go:122] "RemoveContainer" containerID="c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d" Nov 03 14:30:42.965311 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.965290 6389 scope.go:122] "RemoveContainer" containerID="c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d" Nov 03 14:30:42.965642 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:30:42.965590 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d\": container with ID starting with c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d not found: ID does not exist" containerID="c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d" Nov 03 14:30:42.965714 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.965637 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d"} err="failed to get container status \"c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d\": rpc error: code = NotFound desc = could not find container \"c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d\": container with ID starting with c1635843f5591a36efb38f4be6c0b1998b66ec8899d92720e54af619fc0e3d2d not found: ID does not exist" Nov 03 14:30:42.994881 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.994523 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09cd02dc-05d6-47f2-82e3-b5399e421025-kube-api-access-hdtv6\" (UniqueName: \"kubernetes.io/projected/09cd02dc-05d6-47f2-82e3-b5399e421025-kube-api-access-hdtv6\") pod \"09cd02dc-05d6-47f2-82e3-b5399e421025\" (UID: \"09cd02dc-05d6-47f2-82e3-b5399e421025\") " Nov 03 14:30:42.994881 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.994861 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09cd02dc-05d6-47f2-82e3-b5399e421025-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09cd02dc-05d6-47f2-82e3-b5399e421025-plugins-config-volume\") pod \"09cd02dc-05d6-47f2-82e3-b5399e421025\" (UID: \"09cd02dc-05d6-47f2-82e3-b5399e421025\") " Nov 03 14:30:42.995285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.995257 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09cd02dc-05d6-47f2-82e3-b5399e421025-plugins-config-volume" pod "09cd02dc-05d6-47f2-82e3-b5399e421025" (UID: "09cd02dc-05d6-47f2-82e3-b5399e421025"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:42.995405 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.995266 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:30:42.995405 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.995314 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68678292-3210-477f-95bc-3181f870950e-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:30:42.995405 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.995331 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4flmh\" (UniqueName: \"kubernetes.io/projected/68678292-3210-477f-95bc-3181f870950e-kube-api-access-4flmh\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:30:42.995405 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.995347 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68678292-3210-477f-95bc-3181f870950e-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:30:42.997591 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:42.997564 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09cd02dc-05d6-47f2-82e3-b5399e421025-kube-api-access-hdtv6" pod "09cd02dc-05d6-47f2-82e3-b5399e421025" (UID: "09cd02dc-05d6-47f2-82e3-b5399e421025"). InnerVolumeSpecName "kube-api-access-hdtv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:43.053339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.053304 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct"] Nov 03 14:30:43.055725 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.055704 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d608daf1-64d4888778-7tvct"] Nov 03 14:30:43.096523 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.096493 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb01978-2d07-4701-bfa6-81a0eeced7d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb01978-2d07-4701-bfa6-81a0eeced7d3-plugins-config-volume\") pod \"ecb01978-2d07-4701-bfa6-81a0eeced7d3\" (UID: \"ecb01978-2d07-4701-bfa6-81a0eeced7d3\") " Nov 03 14:30:43.096752 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.096682 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb01978-2d07-4701-bfa6-81a0eeced7d3-kube-api-access-27rn6\" (UniqueName: \"kubernetes.io/projected/ecb01978-2d07-4701-bfa6-81a0eeced7d3-kube-api-access-27rn6\") pod \"ecb01978-2d07-4701-bfa6-81a0eeced7d3\" (UID: \"ecb01978-2d07-4701-bfa6-81a0eeced7d3\") " Nov 03 14:30:43.096967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.096940 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb01978-2d07-4701-bfa6-81a0eeced7d3-plugins-config-volume" pod "ecb01978-2d07-4701-bfa6-81a0eeced7d3" (UID: "ecb01978-2d07-4701-bfa6-81a0eeced7d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:43.097319 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.097287 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb01978-2d07-4701-bfa6-81a0eeced7d3-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:30:43.097319 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.097314 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdtv6\" (UniqueName: \"kubernetes.io/projected/09cd02dc-05d6-47f2-82e3-b5399e421025-kube-api-access-hdtv6\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:30:43.097687 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.097329 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09cd02dc-05d6-47f2-82e3-b5399e421025-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:30:43.099349 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.099327 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb01978-2d07-4701-bfa6-81a0eeced7d3-kube-api-access-27rn6" pod "ecb01978-2d07-4701-bfa6-81a0eeced7d3" (UID: "ecb01978-2d07-4701-bfa6-81a0eeced7d3"). InnerVolumeSpecName "kube-api-access-27rn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:43.198520 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.198452 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rn6\" (UniqueName: \"kubernetes.io/projected/ecb01978-2d07-4701-bfa6-81a0eeced7d3-kube-api-access-27rn6\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:30:43.331699 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.331660 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh"] Nov 03 14:30:43.335323 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.335296 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d608daf1-68cb77c885-5l4kh"] Nov 03 14:30:43.484454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.484370 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f"] Nov 03 14:30:43.485111 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.485090 6389 cpu_manager.go:414] "containerMap: removing container" podUID="68678292-3210-477f-95bc-3181f870950e" containerName="maas-api" Nov 03 14:30:43.485236 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.485113 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68678292-3210-477f-95bc-3181f870950e" containerName="maas-api" Nov 03 14:30:43.485236 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.485135 6389 cpu_manager.go:414] "containerMap: removing container" podUID="ecb01978-2d07-4701-bfa6-81a0eeced7d3" containerName="payload-pre-processing" Nov 03 14:30:43.485236 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.485142 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb01978-2d07-4701-bfa6-81a0eeced7d3" containerName="payload-pre-processing" Nov 03 14:30:43.485236 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.485186 6389 cpu_manager.go:414] "containerMap: removing container" podUID="09cd02dc-05d6-47f2-82e3-b5399e421025" containerName="payload-processing" Nov 03 14:30:43.485236 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.485193 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09cd02dc-05d6-47f2-82e3-b5399e421025" containerName="payload-processing" Nov 03 14:30:43.485487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.485347 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb01978-2d07-4701-bfa6-81a0eeced7d3" containerName="payload-pre-processing" Nov 03 14:30:43.485487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.485368 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="09cd02dc-05d6-47f2-82e3-b5399e421025" containerName="payload-processing" Nov 03 14:30:43.485487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.485382 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="68678292-3210-477f-95bc-3181f870950e" containerName="maas-api" Nov 03 14:30:43.535921 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.533244 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f"] Nov 03 14:30:43.535921 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.533396 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.567642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.561537 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b4361f\"" Nov 03 14:30:43.567642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.562423 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-945467806e01\"" Nov 03 14:30:43.567642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.562548 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-e2e--29c79c46-dockercfg-xvjqk\"" Nov 03 14:30:43.567642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.562654 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-95ac97e0ca1e\"" Nov 03 14:30:43.567642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.562713 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-b28d34176fc3\"" Nov 03 14:30:43.567642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.562773 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"openshift-service-ca.crt\"" Nov 03 14:30:43.567642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.562866 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"routing-overlay\"" Nov 03 14:30:43.567642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.562946 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"kube-root-ca.crt\"" Nov 03 14:30:43.649630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.635690 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.649630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.638188 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.649630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.638231 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.649630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.638342 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vrrl\" (UniqueName: \"kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-kube-api-access-6vrrl\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.653755 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.650778 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-c88df00e610c\"" Nov 03 14:30:43.739879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.739788 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vrrl\" (UniqueName: \"kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-kube-api-access-6vrrl\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.739879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.739834 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.740122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.739964 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.740122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.739988 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.740829 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.740799 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.740948 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.740869 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.743118 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.743089 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.761900 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.761868 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vrrl\" (UniqueName: \"kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-kube-api-access-6vrrl\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.947708 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.947679 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:43.962851 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.962826 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" event={"ID":"ecb01978-2d07-4701-bfa6-81a0eeced7d3","Type":"ContainerDied","Data":"8297d4dcdd27318a76b8f0ac9bf043005cd690ddfa9e53a9b37a4e15686da134"} Nov 03 14:30:43.962851 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.962845 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc" Nov 03 14:30:43.963002 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:43.962897 6389 scope.go:122] "RemoveContainer" containerID="04ed5dedb8d97478ad7b5f6bfb67d3cdbf23cf35be9ec914ea4a699a58288483" Nov 03 14:30:44.026532 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.026499 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc"] Nov 03 14:30:44.029912 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.029890 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d608daf1-76c879f85f-msfmc"] Nov 03 14:30:44.256326 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.256299 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f"] Nov 03 14:30:44.257321 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:30:44.257268 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4de2e2_0f1b_49b7_b9db_46a1e3651f4e.slice/crio-8e4798a7cac3c2fcaf62e679c2021fbb4e8283b8ec3446350bed9a791d938113 WatchSource:0}: Error finding container 8e4798a7cac3c2fcaf62e679c2021fbb4e8283b8ec3446350bed9a791d938113: Status 404 returned error can't find the container with id 8e4798a7cac3c2fcaf62e679c2021fbb4e8283b8ec3446350bed9a791d938113 Nov 03 14:30:44.519087 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.519015 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9"] Nov 03 14:30:44.523437 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.523411 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.543271 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.543248 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c1daa315-openshift-default-dockercfg-wg9cf\"" Nov 03 14:30:44.549917 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.549881 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9"] Nov 03 14:30:44.550662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.550641 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-socket\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.550781 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.550679 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e14293-9f59-464b-ab6a-ec832c69b208-istiod-ca-cert\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.550836 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.550810 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-token\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.550879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.550851 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-certs\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.550939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.550895 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-credential-socket\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.550994 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.550957 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-envoy\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.551046 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.551004 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-podinfo\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.551198 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.551182 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g55tt\" (UniqueName: \"kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-kube-api-access-g55tt\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.551253 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.551242 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-data\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.618722 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.618690 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09cd02dc-05d6-47f2-82e3-b5399e421025" path="/var/lib/kubelet/pods/09cd02dc-05d6-47f2-82e3-b5399e421025/volumes" Nov 03 14:30:44.619152 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.619138 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68678292-3210-477f-95bc-3181f870950e" path="/var/lib/kubelet/pods/68678292-3210-477f-95bc-3181f870950e/volumes" Nov 03 14:30:44.619507 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.619495 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb01978-2d07-4701-bfa6-81a0eeced7d3" path="/var/lib/kubelet/pods/ecb01978-2d07-4701-bfa6-81a0eeced7d3/volumes" Nov 03 14:30:44.662870 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.662834 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g55tt\" (UniqueName: \"kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-kube-api-access-g55tt\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.663174 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.663140 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-data\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.684633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.663363 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-socket\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.684926 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.684898 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e14293-9f59-464b-ab6a-ec832c69b208-istiod-ca-cert\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.685108 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.685092 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-token\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.685230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.685213 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-certs\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.685370 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.685351 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-credential-socket\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.685534 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.685517 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-envoy\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.685668 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.685651 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-podinfo\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.686552 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.686528 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-data\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.687073 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.687050 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-socket\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.687849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.687829 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e14293-9f59-464b-ab6a-ec832c69b208-istiod-ca-cert\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.688541 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.688509 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-certs\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.696149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.696127 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-credential-socket\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.738337 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.738306 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g55tt\" (UniqueName: \"kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-kube-api-access-g55tt\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.738856 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.738785 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-envoy\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.739486 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.739465 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-token\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.740102 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.740081 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-podinfo\") pod \"e2e-ait-c1daa315-openshift-default-87b646887-fxnt9\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.852301 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.852224 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:44.982252 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.982218 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" event={"ID":"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e","Type":"ContainerStarted","Data":"18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d"} Nov 03 14:30:44.982441 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.982277 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" event={"ID":"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e","Type":"ContainerStarted","Data":"8e4798a7cac3c2fcaf62e679c2021fbb4e8283b8ec3446350bed9a791d938113"} Nov 03 14:30:44.982763 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:44.982693 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:45.026821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:45.026763 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" podStartSLOduration=2.02674909 podStartE2EDuration="2.02674909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:45.024732517 +0000 UTC m=+1120.583368998" watchObservedRunningTime="2026-11-03 14:30:45.02674909 +0000 UTC m=+1120.585385574" Nov 03 14:30:45.174746 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:45.174710 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9"] Nov 03 14:30:45.178575 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:45.178535 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:30:45.178708 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:45.178653 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:30:45.178777 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:45.178718 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:30:45.989489 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:45.989453 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" event={"ID":"f3e14293-9f59-464b-ab6a-ec832c69b208","Type":"ContainerStarted","Data":"2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b"} Nov 03 14:30:45.989906 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:45.989509 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" event={"ID":"f3e14293-9f59-464b-ab6a-ec832c69b208","Type":"ContainerStarted","Data":"1fdafaa994315545ecec706d193ff08b7cd5097522f4232020225f6d46caac08"} Nov 03 14:30:46.037146 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:46.037089 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" podStartSLOduration=2.037072037 podStartE2EDuration="2.037072037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:46.03639935 +0000 UTC m=+1121.595035837" watchObservedRunningTime="2026-11-03 14:30:46.037072037 +0000 UTC m=+1121.595708523" Nov 03 14:30:46.852931 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:46.852894 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:46.858749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:46.858717 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:46.995858 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:46.995827 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:46.996914 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:46.996893 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:30:50.403047 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.403018 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk"] Nov 03 14:30:50.407172 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.407147 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" Nov 03 14:30:50.425029 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.425009 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c1daa315-dockercfg-hml9s\"" Nov 03 14:30:50.427781 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.427757 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk"] Nov 03 14:30:50.427901 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.427804 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c1daa315\"" Nov 03 14:30:50.514636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.514125 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmhg6\" (UniqueName: \"kubernetes.io/projected/3aa38da9-e68c-4524-aab3-4309d89327ba-kube-api-access-zmhg6\") pod \"payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk\" (UID: \"3aa38da9-e68c-4524-aab3-4309d89327ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" Nov 03 14:30:50.514636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.514289 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa38da9-e68c-4524-aab3-4309d89327ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk\" (UID: \"3aa38da9-e68c-4524-aab3-4309d89327ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" Nov 03 14:30:50.620206 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.620173 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmhg6\" (UniqueName: \"kubernetes.io/projected/3aa38da9-e68c-4524-aab3-4309d89327ba-kube-api-access-zmhg6\") pod \"payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk\" (UID: \"3aa38da9-e68c-4524-aab3-4309d89327ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" Nov 03 14:30:50.620570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.620532 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa38da9-e68c-4524-aab3-4309d89327ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk\" (UID: \"3aa38da9-e68c-4524-aab3-4309d89327ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" Nov 03 14:30:50.622086 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.622066 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa38da9-e68c-4524-aab3-4309d89327ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk\" (UID: \"3aa38da9-e68c-4524-aab3-4309d89327ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" Nov 03 14:30:50.672680 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.671064 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh"] Nov 03 14:30:50.685335 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.685307 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmhg6\" (UniqueName: \"kubernetes.io/projected/3aa38da9-e68c-4524-aab3-4309d89327ba-kube-api-access-zmhg6\") pod \"payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk\" (UID: \"3aa38da9-e68c-4524-aab3-4309d89327ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" Nov 03 14:30:50.686439 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.686415 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh"] Nov 03 14:30:50.686577 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.686507 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" Nov 03 14:30:50.721428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.721403 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" Nov 03 14:30:50.721828 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.721802 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-plugins-config-volume\") pod \"payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh\" (UID: \"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3\") " pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" Nov 03 14:30:50.721952 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.721845 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2czk7\" (UniqueName: \"kubernetes.io/projected/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-kube-api-access-2czk7\") pod \"payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh\" (UID: \"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3\") " pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" Nov 03 14:30:50.822881 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.822849 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-plugins-config-volume\") pod \"payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh\" (UID: \"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3\") " pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" Nov 03 14:30:50.822881 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.822892 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2czk7\" (UniqueName: \"kubernetes.io/projected/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-kube-api-access-2czk7\") pod \"payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh\" (UID: \"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3\") " pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" Nov 03 14:30:50.824062 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.824033 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-plugins-config-volume\") pod \"payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh\" (UID: \"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3\") " pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" Nov 03 14:30:50.845432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:50.845404 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2czk7\" (UniqueName: \"kubernetes.io/projected/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-kube-api-access-2czk7\") pod \"payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh\" (UID: \"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3\") " pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" Nov 03 14:30:51.001345 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:51.001208 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" Nov 03 14:30:51.029683 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:51.029654 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk"] Nov 03 14:30:51.030170 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:30:51.030147 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa38da9_e68c_4524_aab3_4309d89327ba.slice/crio-f76321b154fe1dafffca5d66ae916666e7ce11ad2f016308acd2cd097d819298 WatchSource:0}: Error finding container f76321b154fe1dafffca5d66ae916666e7ce11ad2f016308acd2cd097d819298: Status 404 returned error can't find the container with id f76321b154fe1dafffca5d66ae916666e7ce11ad2f016308acd2cd097d819298 Nov 03 14:30:51.081644 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:51.073982 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" event={"ID":"3aa38da9-e68c-4524-aab3-4309d89327ba","Type":"ContainerStarted","Data":"f76321b154fe1dafffca5d66ae916666e7ce11ad2f016308acd2cd097d819298"} Nov 03 14:30:51.305363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:51.305334 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh"] Nov 03 14:30:51.306523 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:30:51.306498 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f3fd36_ef76_4ba3_9b0d_e5cc41ab2cb3.slice/crio-2fad9cf2c8992b66c552c1cc006032140adc725f48fe13fda5a356987b953c79 WatchSource:0}: Error finding container 2fad9cf2c8992b66c552c1cc006032140adc725f48fe13fda5a356987b953c79: Status 404 returned error can't find the container with id 2fad9cf2c8992b66c552c1cc006032140adc725f48fe13fda5a356987b953c79 Nov 03 14:30:52.083174 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:52.083133 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" event={"ID":"3aa38da9-e68c-4524-aab3-4309d89327ba","Type":"ContainerStarted","Data":"87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b"} Nov 03 14:30:52.083845 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:52.083225 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" Nov 03 14:30:52.084922 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:52.084901 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" event={"ID":"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3","Type":"ContainerStarted","Data":"b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75"} Nov 03 14:30:52.085060 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:52.084936 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" event={"ID":"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3","Type":"ContainerStarted","Data":"2fad9cf2c8992b66c552c1cc006032140adc725f48fe13fda5a356987b953c79"} Nov 03 14:30:52.085060 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:52.085020 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" Nov 03 14:30:52.124733 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:52.124690 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" podStartSLOduration=2.124676576 podStartE2EDuration="2.124676576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:52.124263857 +0000 UTC m=+1127.682915874" watchObservedRunningTime="2026-11-03 14:30:52.124676576 +0000 UTC m=+1127.683313071" Nov 03 14:30:52.174175 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:52.174130 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" podStartSLOduration=2.174118319 podStartE2EDuration="2.174118319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:52.173405009 +0000 UTC m=+1127.732041494" watchObservedRunningTime="2026-11-03 14:30:52.174118319 +0000 UTC m=+1127.732754844" Nov 03 14:30:55.992333 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:30:55.992309 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:30:58.373107 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:30:58.373071 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:31:02.069269 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:02.069235 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d"] Nov 03 14:31:03.129246 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:03.129218 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" Nov 03 14:31:03.141824 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:03.141800 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" Nov 03 14:31:05.603630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.603567 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx"] Nov 03 14:31:05.608898 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.608877 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.631152 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.631116 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-maas-api-tls\") pod \"maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.631339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.631200 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5kl\" (UniqueName: \"kubernetes.io/projected/fb78837b-2cfe-4e57-abac-903686341a47-kube-api-access-ks5kl\") pod \"maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.631339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.631302 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-metrics-certs\") pod \"maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.631691 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.631350 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb78837b-2cfe-4e57-abac-903686341a47-openshift-service-ca\") pod \"maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.637939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.637902 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c1daa315-serving-cert\"" Nov 03 14:31:05.642717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.642689 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx"] Nov 03 14:31:05.732196 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.732159 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-maas-api-tls\") pod \"maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.732386 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.732212 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5kl\" (UniqueName: \"kubernetes.io/projected/fb78837b-2cfe-4e57-abac-903686341a47-kube-api-access-ks5kl\") pod \"maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.732386 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.732242 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-metrics-certs\") pod \"maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.732386 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.732261 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb78837b-2cfe-4e57-abac-903686341a47-openshift-service-ca\") pod \"maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.732947 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.732923 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb78837b-2cfe-4e57-abac-903686341a47-openshift-service-ca\") pod \"maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.734707 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.734685 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-maas-api-tls\") pod \"maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.734759 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.734732 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-metrics-certs\") pod \"maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.757116 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.757090 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5kl\" (UniqueName: \"kubernetes.io/projected/fb78837b-2cfe-4e57-abac-903686341a47-kube-api-access-ks5kl\") pod \"maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:05.925038 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:05.925010 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:06.438115 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:06.438068 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx"] Nov 03 14:31:06.440264 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:31:06.440229 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb78837b_2cfe_4e57_abac_903686341a47.slice/crio-ca25cc46cd460ae66f1dde29e7441856536b077af5714925a7746e5eafe2e975 WatchSource:0}: Error finding container ca25cc46cd460ae66f1dde29e7441856536b077af5714925a7746e5eafe2e975: Status 404 returned error can't find the container with id ca25cc46cd460ae66f1dde29e7441856536b077af5714925a7746e5eafe2e975 Nov 03 14:31:06.442523 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:06.442503 6389 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:31:07.070524 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:07.070485 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" podUID="2dcec4aa-e7d6-4d20-b29b-bf7890788744" containerName="payload-processing" containerID="cri-o://4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a" gracePeriod=25 Nov 03 14:31:07.515800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:07.515236 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" event={"ID":"fb78837b-2cfe-4e57-abac-903686341a47","Type":"ContainerStarted","Data":"a8936ce0180a4bbfcc0635ea33fbf7a4643052b6d484b90a7e20b1c8ddf085a2"} Nov 03 14:31:07.515800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:07.515348 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" event={"ID":"fb78837b-2cfe-4e57-abac-903686341a47","Type":"ContainerStarted","Data":"ca25cc46cd460ae66f1dde29e7441856536b077af5714925a7746e5eafe2e975"} Nov 03 14:31:07.515800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:07.515728 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:07.583636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:07.582646 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" podStartSLOduration=2.057389082 podStartE2EDuration="2.582627141s" totalImagesPullingTime="525.238059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:07.579305426 +0000 UTC m=+1143.137941911" watchObservedRunningTime="2026-11-03 14:31:07.582627141 +0000 UTC m=+1143.141263626" Nov 03 14:31:13.542429 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:13.542401 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:15.497881 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:15.497840 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" podUID="2dcec4aa-e7d6-4d20-b29b-bf7890788744" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:31:17.714538 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.714503 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52"] Nov 03 14:31:17.718963 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.718938 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.754712 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.754681 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52"] Nov 03 14:31:17.820233 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.820187 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.820233 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.820237 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.820479 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.820299 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.820479 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.820332 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.820479 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.820352 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.820603 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.820514 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.820603 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.820539 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.820603 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.820570 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r584j\" (UniqueName: \"kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-kube-api-access-r584j\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.820742 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.820694 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.922212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.922177 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.922392 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.922223 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.922392 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.922243 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.922392 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.922358 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.922565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.922402 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.922660 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.922578 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.922660 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.922639 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.922783 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.922694 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r584j\" (UniqueName: \"kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-kube-api-access-r584j\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.922840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.922792 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.922840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.922805 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.922938 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.922857 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.923033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.923007 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.923080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.923032 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.923297 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.923274 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.925296 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.925272 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.925402 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.925386 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.945179 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.945153 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:17.945276 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:17.945189 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r584j\" (UniqueName: \"kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-kube-api-access-r584j\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:18.032911 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:18.032832 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:18.357411 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:18.357380 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52"] Nov 03 14:31:18.358415 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:31:18.358385 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be2ce35_3ee0_4aa8_88c9_19d23c597ca1.slice/crio-6300aef84c7c97f8d5bac067b3d0ab162b55563929de94a8b3dd38ce0a47cca7 WatchSource:0}: Error finding container 6300aef84c7c97f8d5bac067b3d0ab162b55563929de94a8b3dd38ce0a47cca7: Status 404 returned error can't find the container with id 6300aef84c7c97f8d5bac067b3d0ab162b55563929de94a8b3dd38ce0a47cca7 Nov 03 14:31:18.360789 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:18.360760 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:31:18.360862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:18.360822 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:31:18.360862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:18.360850 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:31:18.603211 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:18.600753 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" event={"ID":"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1","Type":"ContainerStarted","Data":"c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b"} Nov 03 14:31:18.603211 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:18.600830 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" event={"ID":"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1","Type":"ContainerStarted","Data":"6300aef84c7c97f8d5bac067b3d0ab162b55563929de94a8b3dd38ce0a47cca7"} Nov 03 14:31:18.651363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:18.651203 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" podStartSLOduration=1.651184739 podStartE2EDuration="1.651184739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:18.648899765 +0000 UTC m=+1154.207536273" watchObservedRunningTime="2026-11-03 14:31:18.651184739 +0000 UTC m=+1154.209821224" Nov 03 14:31:19.033494 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:19.033459 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:19.035212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:19.035190 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 14:31:19.035279 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:19.035243 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" podUID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 14:31:20.044136 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:20.044096 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 14:31:20.044589 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:20.044177 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" podUID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 14:31:21.041179 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:21.041137 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 14:31:21.041477 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:21.041442 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" podUID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 14:31:22.033310 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:22.033275 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 14:31:22.033990 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:22.033950 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" podUID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 14:31:23.037638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:23.035082 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 14:31:23.037638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:23.035149 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" podUID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 14:31:24.041672 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:24.039033 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 14:31:24.041672 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:24.039093 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" podUID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 14:31:25.046634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:25.046057 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 14:31:25.046634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:25.046118 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" podUID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 14:31:25.497257 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:25.497210 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" podUID="2dcec4aa-e7d6-4d20-b29b-bf7890788744" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:31:26.430145 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:26.430111 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:26.432847 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:26.432823 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:26.453300 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:26.452514 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:31:27.395438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.395405 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" Nov 03 14:31:27.473166 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.473135 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dcec4aa-e7d6-4d20-b29b-bf7890788744-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dcec4aa-e7d6-4d20-b29b-bf7890788744-plugins-config-volume\") pod \"2dcec4aa-e7d6-4d20-b29b-bf7890788744\" (UID: \"2dcec4aa-e7d6-4d20-b29b-bf7890788744\") " Nov 03 14:31:27.473732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.473271 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dcec4aa-e7d6-4d20-b29b-bf7890788744-kube-api-access-t8w2b\" (UniqueName: \"kubernetes.io/projected/2dcec4aa-e7d6-4d20-b29b-bf7890788744-kube-api-access-t8w2b\") pod \"2dcec4aa-e7d6-4d20-b29b-bf7890788744\" (UID: \"2dcec4aa-e7d6-4d20-b29b-bf7890788744\") " Nov 03 14:31:27.474257 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.474227 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcec4aa-e7d6-4d20-b29b-bf7890788744-plugins-config-volume" pod "2dcec4aa-e7d6-4d20-b29b-bf7890788744" (UID: "2dcec4aa-e7d6-4d20-b29b-bf7890788744"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:27.476842 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.476807 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcec4aa-e7d6-4d20-b29b-bf7890788744-kube-api-access-t8w2b" pod "2dcec4aa-e7d6-4d20-b29b-bf7890788744" (UID: "2dcec4aa-e7d6-4d20-b29b-bf7890788744"). InnerVolumeSpecName "kube-api-access-t8w2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:27.575169 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.575132 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8w2b\" (UniqueName: \"kubernetes.io/projected/2dcec4aa-e7d6-4d20-b29b-bf7890788744-kube-api-access-t8w2b\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:27.575312 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.575177 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dcec4aa-e7d6-4d20-b29b-bf7890788744-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:27.807729 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.807693 6389 generic.go:356] "Generic (PLEG): container finished" podID="2dcec4aa-e7d6-4d20-b29b-bf7890788744" containerID="4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a" exitCode=0 Nov 03 14:31:27.807903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.807735 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" event={"ID":"2dcec4aa-e7d6-4d20-b29b-bf7890788744","Type":"ContainerDied","Data":"4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a"} Nov 03 14:31:27.807903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.807758 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" Nov 03 14:31:27.807903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.807786 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d" event={"ID":"2dcec4aa-e7d6-4d20-b29b-bf7890788744","Type":"ContainerDied","Data":"e82e18df22425da7bab6fa69be55e4a5b35ba1960dfa2e97fb2ea970f3d0a712"} Nov 03 14:31:27.807903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.807808 6389 scope.go:122] "RemoveContainer" containerID="4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a" Nov 03 14:31:27.821475 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.821458 6389 scope.go:122] "RemoveContainer" containerID="4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a" Nov 03 14:31:27.821763 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:31:27.821745 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a\": container with ID starting with 4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a not found: ID does not exist" containerID="4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a" Nov 03 14:31:27.821822 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.821771 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a"} err="failed to get container status \"4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a\": rpc error: code = NotFound desc = could not find container \"4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a\": container with ID starting with 4060712d3afac1f268136a02e2b354f354accc7a6543503f319057ce7629f06a not found: ID does not exist" Nov 03 14:31:27.864445 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.864405 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d"] Nov 03 14:31:27.878962 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:27.878905 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-h9p4d"] Nov 03 14:31:28.377851 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:31:28.377806 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:31:28.619464 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:28.619432 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dcec4aa-e7d6-4d20-b29b-bf7890788744" path="/var/lib/kubelet/pods/2dcec4aa-e7d6-4d20-b29b-bf7890788744/volumes" Nov 03 14:31:31.864470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:31.862973 6389 generic.go:356] "Generic (PLEG): container finished" podID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerID="2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b" exitCode=0 Nov 03 14:31:31.864470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:31.863121 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" event={"ID":"f3e14293-9f59-464b-ab6a-ec832c69b208","Type":"ContainerDied","Data":"2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b"} Nov 03 14:31:31.864470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:31.863652 6389 scope.go:122] "RemoveContainer" containerID="2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b" Nov 03 14:31:31.867649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:31.867110 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:31:31.867649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:31.867211 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:31:31.867649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:31.867261 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:31:31.995838 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:31.995811 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:31:32.988635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:32.972040 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" event={"ID":"f3e14293-9f59-464b-ab6a-ec832c69b208","Type":"ContainerStarted","Data":"c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38"} Nov 03 14:31:33.880635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:33.860350 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:31:33.880635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:33.870325 6389 patch_prober.go:29] interesting pod/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 14:31:33.880635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:33.870383 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 14:31:34.079634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:34.078725 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:31:34.079634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:34.078792 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:31:34.853207 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:34.853171 6389 patch_prober.go:29] interesting pod/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 14:31:34.853397 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:34.853224 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 14:31:35.852754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:35.852718 6389 patch_prober.go:29] interesting pod/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 14:31:35.853152 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:35.852771 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 14:31:36.853285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:36.853247 6389 patch_prober.go:29] interesting pod/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 14:31:36.853753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:36.853331 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 14:31:37.853395 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:37.853347 6389 patch_prober.go:29] interesting pod/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 14:31:37.853941 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:37.853714 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 14:31:38.858110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:38.858074 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:31:38.858651 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:38.858492 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:31:38.859121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:38.859104 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:31:40.040272 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:40.040238 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh"] Nov 03 14:31:40.094543 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:40.094513 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk"] Nov 03 14:31:41.962420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:41.962383 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx"] Nov 03 14:31:41.962962 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:41.962691 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" podUID="fb78837b-2cfe-4e57-abac-903686341a47" containerName="maas-api" containerID="cri-o://a8936ce0180a4bbfcc0635ea33fbf7a4643052b6d484b90a7e20b1c8ddf085a2" gracePeriod=30 Nov 03 14:31:42.117848 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.117809 6389 generic.go:356] "Generic (PLEG): container finished" podID="fb78837b-2cfe-4e57-abac-903686341a47" containerID="a8936ce0180a4bbfcc0635ea33fbf7a4643052b6d484b90a7e20b1c8ddf085a2" exitCode=0 Nov 03 14:31:42.118021 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.117874 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" event={"ID":"fb78837b-2cfe-4e57-abac-903686341a47","Type":"ContainerDied","Data":"a8936ce0180a4bbfcc0635ea33fbf7a4643052b6d484b90a7e20b1c8ddf085a2"} Nov 03 14:31:42.298334 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.297999 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:42.366000 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.365963 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-maas-api-tls\") pod \"fb78837b-2cfe-4e57-abac-903686341a47\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " Nov 03 14:31:42.368044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.368018 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb78837b-2cfe-4e57-abac-903686341a47-openshift-service-ca" pod "fb78837b-2cfe-4e57-abac-903686341a47" (UID: "fb78837b-2cfe-4e57-abac-903686341a47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:42.368226 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.367507 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb78837b-2cfe-4e57-abac-903686341a47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb78837b-2cfe-4e57-abac-903686341a47-openshift-service-ca\") pod \"fb78837b-2cfe-4e57-abac-903686341a47\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " Nov 03 14:31:42.368860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.368393 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-metrics-certs\") pod \"fb78837b-2cfe-4e57-abac-903686341a47\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " Nov 03 14:31:42.369454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.369436 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb78837b-2cfe-4e57-abac-903686341a47-kube-api-access-ks5kl\" (UniqueName: \"kubernetes.io/projected/fb78837b-2cfe-4e57-abac-903686341a47-kube-api-access-ks5kl\") pod \"fb78837b-2cfe-4e57-abac-903686341a47\" (UID: \"fb78837b-2cfe-4e57-abac-903686341a47\") " Nov 03 14:31:42.373307 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.373266 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb78837b-2cfe-4e57-abac-903686341a47-kube-api-access-ks5kl" pod "fb78837b-2cfe-4e57-abac-903686341a47" (UID: "fb78837b-2cfe-4e57-abac-903686341a47"). InnerVolumeSpecName "kube-api-access-ks5kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:42.381696 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.381602 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb78837b-2cfe-4e57-abac-903686341a47-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:42.381696 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.381655 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks5kl\" (UniqueName: \"kubernetes.io/projected/fb78837b-2cfe-4e57-abac-903686341a47-kube-api-access-ks5kl\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:42.393890 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.393840 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-metrics-certs" pod "fb78837b-2cfe-4e57-abac-903686341a47" (UID: "fb78837b-2cfe-4e57-abac-903686341a47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:31:42.394957 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.394729 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-maas-api-tls" pod "fb78837b-2cfe-4e57-abac-903686341a47" (UID: "fb78837b-2cfe-4e57-abac-903686341a47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:31:42.466999 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.466855 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb"] Nov 03 14:31:42.467454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.467381 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" podUID="9738db50-dced-4ca1-8ab0-305ac60d8e8d" containerName="maas-api" containerID="cri-o://cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d" gracePeriod=30 Nov 03 14:31:42.483393 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.483237 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:42.484258 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.483574 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb78837b-2cfe-4e57-abac-903686341a47-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:42.731011 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.730985 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:31:42.788025 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.787990 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-metrics-certs\") pod \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " Nov 03 14:31:42.788204 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.788070 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9738db50-dced-4ca1-8ab0-305ac60d8e8d-kube-api-access-2prmh\" (UniqueName: \"kubernetes.io/projected/9738db50-dced-4ca1-8ab0-305ac60d8e8d-kube-api-access-2prmh\") pod \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " Nov 03 14:31:42.788204 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.788135 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9738db50-dced-4ca1-8ab0-305ac60d8e8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9738db50-dced-4ca1-8ab0-305ac60d8e8d-openshift-service-ca\") pod \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " Nov 03 14:31:42.788364 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.788347 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-maas-api-tls\") pod \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\" (UID: \"9738db50-dced-4ca1-8ab0-305ac60d8e8d\") " Nov 03 14:31:42.788766 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.788504 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9738db50-dced-4ca1-8ab0-305ac60d8e8d-openshift-service-ca" pod "9738db50-dced-4ca1-8ab0-305ac60d8e8d" (UID: "9738db50-dced-4ca1-8ab0-305ac60d8e8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:42.788974 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.788957 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9738db50-dced-4ca1-8ab0-305ac60d8e8d-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:42.790740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.790719 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-metrics-certs" pod "9738db50-dced-4ca1-8ab0-305ac60d8e8d" (UID: "9738db50-dced-4ca1-8ab0-305ac60d8e8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:31:42.790922 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.790905 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-maas-api-tls" pod "9738db50-dced-4ca1-8ab0-305ac60d8e8d" (UID: "9738db50-dced-4ca1-8ab0-305ac60d8e8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:31:42.790982 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.790966 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9738db50-dced-4ca1-8ab0-305ac60d8e8d-kube-api-access-2prmh" pod "9738db50-dced-4ca1-8ab0-305ac60d8e8d" (UID: "9738db50-dced-4ca1-8ab0-305ac60d8e8d"). InnerVolumeSpecName "kube-api-access-2prmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:42.891130 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.891089 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:42.891304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.891173 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2prmh\" (UniqueName: \"kubernetes.io/projected/9738db50-dced-4ca1-8ab0-305ac60d8e8d-kube-api-access-2prmh\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:42.891304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:42.891192 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9738db50-dced-4ca1-8ab0-305ac60d8e8d-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:43.127148 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.127114 6389 generic.go:356] "Generic (PLEG): container finished" podID="9738db50-dced-4ca1-8ab0-305ac60d8e8d" containerID="cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d" exitCode=0 Nov 03 14:31:43.127642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.127194 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" event={"ID":"9738db50-dced-4ca1-8ab0-305ac60d8e8d","Type":"ContainerDied","Data":"cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d"} Nov 03 14:31:43.127642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.127257 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" event={"ID":"9738db50-dced-4ca1-8ab0-305ac60d8e8d","Type":"ContainerDied","Data":"103ec2b3a4101b67da33afc8c01c3d4b4d51576a75e961b6bd47ed5f0e5b483c"} Nov 03 14:31:43.127642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.127290 6389 scope.go:122] "RemoveContainer" containerID="cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d" Nov 03 14:31:43.127642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.127402 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb" Nov 03 14:31:43.129032 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.129014 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" Nov 03 14:31:43.129032 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.129010 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx" event={"ID":"fb78837b-2cfe-4e57-abac-903686341a47","Type":"ContainerDied","Data":"ca25cc46cd460ae66f1dde29e7441856536b077af5714925a7746e5eafe2e975"} Nov 03 14:31:43.139299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.139009 6389 scope.go:122] "RemoveContainer" containerID="cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d" Nov 03 14:31:43.139431 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:31:43.139366 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d\": container with ID starting with cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d not found: ID does not exist" containerID="cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d" Nov 03 14:31:43.139431 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.139391 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d"} err="failed to get container status \"cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d\": rpc error: code = NotFound desc = could not find container \"cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d\": container with ID starting with cf62a75aa04a8ac8e9f170acc03adb5495030e818b0ce4ff77839f8dc7ff1b1d not found: ID does not exist" Nov 03 14:31:43.139431 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.139415 6389 scope.go:122] "RemoveContainer" containerID="a8936ce0180a4bbfcc0635ea33fbf7a4643052b6d484b90a7e20b1c8ddf085a2" Nov 03 14:31:43.188793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.188765 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx"] Nov 03 14:31:43.201476 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.201452 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1daa315-5b5b88dc5b-pvrcx"] Nov 03 14:31:43.236527 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.236489 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb"] Nov 03 14:31:43.240656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:43.240631 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-768cb"] Nov 03 14:31:44.619342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:44.619312 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9738db50-dced-4ca1-8ab0-305ac60d8e8d" path="/var/lib/kubelet/pods/9738db50-dced-4ca1-8ab0-305ac60d8e8d/volumes" Nov 03 14:31:44.619928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:44.619904 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb78837b-2cfe-4e57-abac-903686341a47" path="/var/lib/kubelet/pods/fb78837b-2cfe-4e57-abac-903686341a47/volumes" Nov 03 14:31:44.822750 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:44.822721 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9"] Nov 03 14:31:44.823554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:44.823519 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerName="istio-proxy" containerID="cri-o://c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38" gracePeriod=30 Nov 03 14:31:45.041560 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.041513 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" podUID="82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3" containerName="payload-processing" containerID="cri-o://b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75" gracePeriod=25 Nov 03 14:31:45.104155 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.095756 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" podUID="3aa38da9-e68c-4524-aab3-4309d89327ba" containerName="payload-pre-processing" containerID="cri-o://87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b" gracePeriod=25 Nov 03 14:31:45.444905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.444883 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" Nov 03 14:31:45.476377 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.476354 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" Nov 03 14:31:45.562461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.562372 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-plugins-config-volume\") pod \"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3\" (UID: \"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3\") " Nov 03 14:31:45.562461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.562438 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa38da9-e68c-4524-aab3-4309d89327ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa38da9-e68c-4524-aab3-4309d89327ba-plugins-config-volume\") pod \"3aa38da9-e68c-4524-aab3-4309d89327ba\" (UID: \"3aa38da9-e68c-4524-aab3-4309d89327ba\") " Nov 03 14:31:45.562461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.562462 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa38da9-e68c-4524-aab3-4309d89327ba-kube-api-access-zmhg6\" (UniqueName: \"kubernetes.io/projected/3aa38da9-e68c-4524-aab3-4309d89327ba-kube-api-access-zmhg6\") pod \"3aa38da9-e68c-4524-aab3-4309d89327ba\" (UID: \"3aa38da9-e68c-4524-aab3-4309d89327ba\") " Nov 03 14:31:45.562761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.562511 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-kube-api-access-2czk7\" (UniqueName: \"kubernetes.io/projected/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-kube-api-access-2czk7\") pod \"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3\" (UID: \"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3\") " Nov 03 14:31:45.562896 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.562874 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-plugins-config-volume" pod "82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3" (UID: "82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:45.562975 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.562932 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa38da9-e68c-4524-aab3-4309d89327ba-plugins-config-volume" pod "3aa38da9-e68c-4524-aab3-4309d89327ba" (UID: "3aa38da9-e68c-4524-aab3-4309d89327ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:45.563110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.563088 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:45.563110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.563111 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa38da9-e68c-4524-aab3-4309d89327ba-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:45.565474 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.565452 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa38da9-e68c-4524-aab3-4309d89327ba-kube-api-access-zmhg6" pod "3aa38da9-e68c-4524-aab3-4309d89327ba" (UID: "3aa38da9-e68c-4524-aab3-4309d89327ba"). InnerVolumeSpecName "kube-api-access-zmhg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:45.565629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.565590 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-kube-api-access-2czk7" pod "82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3" (UID: "82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3"). InnerVolumeSpecName "kube-api-access-2czk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:45.665523 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.665490 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmhg6\" (UniqueName: \"kubernetes.io/projected/3aa38da9-e68c-4524-aab3-4309d89327ba-kube-api-access-zmhg6\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:45.665523 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:45.665518 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2czk7\" (UniqueName: \"kubernetes.io/projected/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3-kube-api-access-2czk7\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:46.116630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.116236 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw"] Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117096 6389 cpu_manager.go:414] "containerMap: removing container" podUID="9738db50-dced-4ca1-8ab0-305ac60d8e8d" containerName="maas-api" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117118 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9738db50-dced-4ca1-8ab0-305ac60d8e8d" containerName="maas-api" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117145 6389 cpu_manager.go:414] "containerMap: removing container" podUID="82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3" containerName="payload-processing" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117153 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3" containerName="payload-processing" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117185 6389 cpu_manager.go:414] "containerMap: removing container" podUID="3aa38da9-e68c-4524-aab3-4309d89327ba" containerName="payload-pre-processing" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117193 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa38da9-e68c-4524-aab3-4309d89327ba" containerName="payload-pre-processing" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117207 6389 cpu_manager.go:414] "containerMap: removing container" podUID="fb78837b-2cfe-4e57-abac-903686341a47" containerName="maas-api" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117214 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb78837b-2cfe-4e57-abac-903686341a47" containerName="maas-api" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117236 6389 cpu_manager.go:414] "containerMap: removing container" podUID="2dcec4aa-e7d6-4d20-b29b-bf7890788744" containerName="payload-processing" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117243 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dcec4aa-e7d6-4d20-b29b-bf7890788744" containerName="payload-processing" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117384 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="9738db50-dced-4ca1-8ab0-305ac60d8e8d" containerName="maas-api" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117395 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3" containerName="payload-processing" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117411 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa38da9-e68c-4524-aab3-4309d89327ba" containerName="payload-pre-processing" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117419 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb78837b-2cfe-4e57-abac-903686341a47" containerName="maas-api" Nov 03 14:31:46.141325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.117429 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dcec4aa-e7d6-4d20-b29b-bf7890788744" containerName="payload-processing" Nov 03 14:31:46.165668 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.143792 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.165668 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.153847 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw"] Nov 03 14:31:46.165668 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.164131 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-09d577dc-openshift-default-dockercfg-ngt4h\"" Nov 03 14:31:46.282841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.279799 6389 generic.go:356] "Generic (PLEG): container finished" podID="82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3" containerID="b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75" exitCode=0 Nov 03 14:31:46.282841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.279990 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" event={"ID":"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3","Type":"ContainerDied","Data":"b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75"} Nov 03 14:31:46.282841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.280040 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" event={"ID":"82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3","Type":"ContainerDied","Data":"2fad9cf2c8992b66c552c1cc006032140adc725f48fe13fda5a356987b953c79"} Nov 03 14:31:46.282841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.280072 6389 scope.go:122] "RemoveContainer" containerID="b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75" Nov 03 14:31:46.282841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.280226 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh" Nov 03 14:31:46.315928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.314255 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-data\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.315928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.314313 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-podinfo\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.315928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.314424 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sszsb\" (UniqueName: \"kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-kube-api-access-sszsb\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.315928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.314457 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-socket\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.315928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.314496 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-envoy\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.315928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.314627 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-credential-socket\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.315928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.314717 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-token\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.315928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.314742 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istiod-ca-cert\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.315928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.314767 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-certs\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.325017 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.318351 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" event={"ID":"3aa38da9-e68c-4524-aab3-4309d89327ba","Type":"ContainerDied","Data":"87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b"} Nov 03 14:31:46.325017 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.318446 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" Nov 03 14:31:46.325017 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.319125 6389 generic.go:356] "Generic (PLEG): container finished" podID="3aa38da9-e68c-4524-aab3-4309d89327ba" containerID="87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b" exitCode=0 Nov 03 14:31:46.325017 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.319214 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk" event={"ID":"3aa38da9-e68c-4524-aab3-4309d89327ba","Type":"ContainerDied","Data":"f76321b154fe1dafffca5d66ae916666e7ce11ad2f016308acd2cd097d819298"} Nov 03 14:31:46.358380 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.358356 6389 scope.go:122] "RemoveContainer" containerID="b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75" Nov 03 14:31:46.358805 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:31:46.358776 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75\": container with ID starting with b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75 not found: ID does not exist" containerID="b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75" Nov 03 14:31:46.358938 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.358817 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75"} err="failed to get container status \"b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75\": rpc error: code = NotFound desc = could not find container \"b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75\": container with ID starting with b740fc7c1578fa6b2d39ae358479fd5393215d3efd1c10b51514ba433367ae75 not found: ID does not exist" Nov 03 14:31:46.358938 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.358843 6389 scope.go:122] "RemoveContainer" containerID="87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b" Nov 03 14:31:46.368418 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.368402 6389 scope.go:122] "RemoveContainer" containerID="87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b" Nov 03 14:31:46.368814 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:31:46.368792 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b\": container with ID starting with 87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b not found: ID does not exist" containerID="87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b" Nov 03 14:31:46.368883 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.368824 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b"} err="failed to get container status \"87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b\": rpc error: code = NotFound desc = could not find container \"87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b\": container with ID starting with 87a3bbcbcd07954ec4452a254fbae015b5b3ed2385b45ba744d7d3951f39096b not found: ID does not exist" Nov 03 14:31:46.389541 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.389515 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh"] Nov 03 14:31:46.397248 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.397223 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1daa315-55d997c78f-5hljh"] Nov 03 14:31:46.416570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.416544 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-token\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.416707 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.416575 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istiod-ca-cert\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.416707 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.416597 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-certs\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.416707 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.416691 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-data\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.416841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.416723 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-podinfo\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.416841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.416763 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sszsb\" (UniqueName: \"kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-kube-api-access-sszsb\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.416841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.416784 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-socket\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.416841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.416818 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-envoy\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.417035 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.416910 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-credential-socket\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.417091 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.417060 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-certs\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.417175 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.417144 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-data\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.417304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.417283 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-socket\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.417339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.417293 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-credential-socket\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.417662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.417639 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istiod-ca-cert\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.419264 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.419241 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-envoy\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.419488 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.419472 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-podinfo\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.434369 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.434347 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk"] Nov 03 14:31:46.444572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.444546 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sszsb\" (UniqueName: \"kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-kube-api-access-sszsb\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.444711 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.444554 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-token\") pod \"e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.448875 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.448856 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1daa315-8689b9b6f5-qpnwk"] Nov 03 14:31:46.481679 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.480814 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:46.623378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.623303 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa38da9-e68c-4524-aab3-4309d89327ba" path="/var/lib/kubelet/pods/3aa38da9-e68c-4524-aab3-4309d89327ba/volumes" Nov 03 14:31:46.623714 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.623700 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3" path="/var/lib/kubelet/pods/82f3fd36-ef76-4ba3-9b0d-e5cc41ab2cb3/volumes" Nov 03 14:31:46.851641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.847967 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw"] Nov 03 14:31:46.882075 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.880370 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:31:46.882075 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.880472 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:31:46.882075 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:46.880516 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:31:47.346369 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:47.346324 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" event={"ID":"3816b61f-c6dc-4b2f-9bb4-f679aa95285f","Type":"ContainerStarted","Data":"37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8"} Nov 03 14:31:47.346554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:47.346431 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" event={"ID":"3816b61f-c6dc-4b2f-9bb4-f679aa95285f","Type":"ContainerStarted","Data":"a03c485748f04f33abde21d5805d28c5f0336970235760c3cc3e121ed7199755"} Nov 03 14:31:47.402519 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:47.402456 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" podStartSLOduration=1.40244233 podStartE2EDuration="1.40244233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:47.399216659 +0000 UTC m=+1182.957853145" watchObservedRunningTime="2026-11-03 14:31:47.40244233 +0000 UTC m=+1182.961078815" Nov 03 14:31:47.481240 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:47.481192 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:48.499633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:48.499264 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:49.067459 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:49.066782 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:31:49.067459 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:49.066862 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:31:49.391807 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:49.391727 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:49.392797 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:49.392774 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:31:50.101074 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.101018 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:31:50.282119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282091 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-socket\") pod \"f3e14293-9f59-464b-ab6a-ec832c69b208\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " Nov 03 14:31:50.282305 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282133 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e14293-9f59-464b-ab6a-ec832c69b208-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e14293-9f59-464b-ab6a-ec832c69b208-istiod-ca-cert\") pod \"f3e14293-9f59-464b-ab6a-ec832c69b208\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " Nov 03 14:31:50.282305 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282152 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-kube-api-access-g55tt\" (UniqueName: \"kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-kube-api-access-g55tt\") pod \"f3e14293-9f59-464b-ab6a-ec832c69b208\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " Nov 03 14:31:50.282305 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282197 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-credential-socket\") pod \"f3e14293-9f59-464b-ab6a-ec832c69b208\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " Nov 03 14:31:50.282305 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282245 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-envoy\") pod \"f3e14293-9f59-464b-ab6a-ec832c69b208\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " Nov 03 14:31:50.282305 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282284 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-podinfo\") pod \"f3e14293-9f59-464b-ab6a-ec832c69b208\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " Nov 03 14:31:50.282567 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282311 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-data\") pod \"f3e14293-9f59-464b-ab6a-ec832c69b208\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " Nov 03 14:31:50.282567 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282370 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-certs\") pod \"f3e14293-9f59-464b-ab6a-ec832c69b208\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " Nov 03 14:31:50.282567 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282419 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-token\" (UniqueName: \"kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-token\") pod \"f3e14293-9f59-464b-ab6a-ec832c69b208\" (UID: \"f3e14293-9f59-464b-ab6a-ec832c69b208\") " Nov 03 14:31:50.282860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282413 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-socket" pod "f3e14293-9f59-464b-ab6a-ec832c69b208" (UID: "f3e14293-9f59-464b-ab6a-ec832c69b208"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:50.282860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282632 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-data" pod "f3e14293-9f59-464b-ab6a-ec832c69b208" (UID: "f3e14293-9f59-464b-ab6a-ec832c69b208"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:50.282860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282675 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e14293-9f59-464b-ab6a-ec832c69b208-istiod-ca-cert" pod "f3e14293-9f59-464b-ab6a-ec832c69b208" (UID: "f3e14293-9f59-464b-ab6a-ec832c69b208"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:50.282860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282804 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-credential-socket" pod "f3e14293-9f59-464b-ab6a-ec832c69b208" (UID: "f3e14293-9f59-464b-ab6a-ec832c69b208"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:50.282860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.282818 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-certs" pod "f3e14293-9f59-464b-ab6a-ec832c69b208" (UID: "f3e14293-9f59-464b-ab6a-ec832c69b208"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:50.283223 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.283205 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:50.283298 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.283229 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:50.283298 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.283243 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:50.283298 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.283257 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e14293-9f59-464b-ab6a-ec832c69b208-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:50.283298 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.283270 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:50.284658 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.284630 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-podinfo" pod "f3e14293-9f59-464b-ab6a-ec832c69b208" (UID: "f3e14293-9f59-464b-ab6a-ec832c69b208"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:31:50.284750 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.284696 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-kube-api-access-g55tt" pod "f3e14293-9f59-464b-ab6a-ec832c69b208" (UID: "f3e14293-9f59-464b-ab6a-ec832c69b208"). InnerVolumeSpecName "kube-api-access-g55tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:50.285167 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.285147 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-token" pod "f3e14293-9f59-464b-ab6a-ec832c69b208" (UID: "f3e14293-9f59-464b-ab6a-ec832c69b208"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:50.285167 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.285147 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-envoy" pod "f3e14293-9f59-464b-ab6a-ec832c69b208" (UID: "f3e14293-9f59-464b-ab6a-ec832c69b208"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:50.384679 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.384571 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g55tt\" (UniqueName: \"kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-kube-api-access-g55tt\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:50.384679 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.384602 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:50.384679 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.384639 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:50.384679 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.384647 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e14293-9f59-464b-ab6a-ec832c69b208-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:31:50.399376 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.399342 6389 generic.go:356] "Generic (PLEG): container finished" podID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerID="c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38" exitCode=0 Nov 03 14:31:50.399555 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.399383 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" event={"ID":"f3e14293-9f59-464b-ab6a-ec832c69b208","Type":"ContainerDied","Data":"c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38"} Nov 03 14:31:50.399555 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.399415 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" Nov 03 14:31:50.399555 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.399459 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9" event={"ID":"f3e14293-9f59-464b-ab6a-ec832c69b208","Type":"ContainerDied","Data":"1fdafaa994315545ecec706d193ff08b7cd5097522f4232020225f6d46caac08"} Nov 03 14:31:50.399555 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.399490 6389 scope.go:122] "RemoveContainer" containerID="c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38" Nov 03 14:31:50.410287 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.410264 6389 scope.go:122] "RemoveContainer" containerID="2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b" Nov 03 14:31:50.419126 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.419105 6389 scope.go:122] "RemoveContainer" containerID="c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38" Nov 03 14:31:50.419411 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:31:50.419394 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38\": container with ID starting with c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38 not found: ID does not exist" containerID="c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38" Nov 03 14:31:50.419466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.419420 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38"} err="failed to get container status \"c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38\": rpc error: code = NotFound desc = could not find container \"c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38\": container with ID starting with c8904e83eb5785fe9a83bd9cc87acb510ce93801689a55b6e2ecc1b3ac2c3a38 not found: ID does not exist" Nov 03 14:31:50.419466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.419440 6389 scope.go:122] "RemoveContainer" containerID="2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b" Nov 03 14:31:50.419719 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:31:50.419702 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b\": container with ID starting with 2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b not found: ID does not exist" containerID="2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b" Nov 03 14:31:50.419782 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.419725 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b"} err="failed to get container status \"2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b\": rpc error: code = NotFound desc = could not find container \"2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b\": container with ID starting with 2fbd0fc15fceac94c4c4cc0c9ed04af995ef68e450df6c1f11cf6d338c94540b not found: ID does not exist" Nov 03 14:31:50.456295 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.456271 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9"] Nov 03 14:31:50.462290 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.462268 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c1daa315-openshift-default-87b646887-fxnt9"] Nov 03 14:31:50.619224 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.619195 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" path="/var/lib/kubelet/pods/f3e14293-9f59-464b-ab6a-ec832c69b208/volumes" Nov 03 14:31:50.996088 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.996055 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr"] Nov 03 14:31:50.996864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.996844 6389 cpu_manager.go:414] "containerMap: removing container" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerName="istio-proxy" Nov 03 14:31:50.996907 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.996869 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerName="istio-proxy" Nov 03 14:31:50.996956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.996931 6389 cpu_manager.go:414] "containerMap: removing container" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerName="istio-proxy" Nov 03 14:31:50.996956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.996938 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerName="istio-proxy" Nov 03 14:31:50.997061 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:50.997050 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerName="istio-proxy" Nov 03 14:31:51.002351 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.002323 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" Nov 03 14:31:51.021237 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.021202 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr"] Nov 03 14:31:51.025682 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.025655 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-09d577dc\"" Nov 03 14:31:51.025925 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.025904 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-09d577dc-dockercfg-wgjh6\"" Nov 03 14:31:51.156642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.152551 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95dedc3-f6d4-417f-9bca-26052f62191c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr\" (UID: \"b95dedc3-f6d4-417f-9bca-26052f62191c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" Nov 03 14:31:51.156642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.152898 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zcm6\" (UniqueName: \"kubernetes.io/projected/b95dedc3-f6d4-417f-9bca-26052f62191c-kube-api-access-8zcm6\") pod \"payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr\" (UID: \"b95dedc3-f6d4-417f-9bca-26052f62191c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" Nov 03 14:31:51.202638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.193452 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm"] Nov 03 14:31:51.202638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.194468 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e14293-9f59-464b-ab6a-ec832c69b208" containerName="istio-proxy" Nov 03 14:31:51.265723 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.245738 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm"] Nov 03 14:31:51.265723 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.245868 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" Nov 03 14:31:51.267148 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.267090 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zcm6\" (UniqueName: \"kubernetes.io/projected/b95dedc3-f6d4-417f-9bca-26052f62191c-kube-api-access-8zcm6\") pod \"payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr\" (UID: \"b95dedc3-f6d4-417f-9bca-26052f62191c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" Nov 03 14:31:51.267403 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.267363 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95dedc3-f6d4-417f-9bca-26052f62191c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr\" (UID: \"b95dedc3-f6d4-417f-9bca-26052f62191c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" Nov 03 14:31:51.267802 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.267769 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnhdb\" (UniqueName: \"kubernetes.io/projected/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-kube-api-access-tnhdb\") pod \"payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm\" (UID: \"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1\") " pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" Nov 03 14:31:51.267954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.267928 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-plugins-config-volume\") pod \"payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm\" (UID: \"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1\") " pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" Nov 03 14:31:51.269446 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.269409 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95dedc3-f6d4-417f-9bca-26052f62191c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr\" (UID: \"b95dedc3-f6d4-417f-9bca-26052f62191c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" Nov 03 14:31:51.366002 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.361269 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zcm6\" (UniqueName: \"kubernetes.io/projected/b95dedc3-f6d4-417f-9bca-26052f62191c-kube-api-access-8zcm6\") pod \"payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr\" (UID: \"b95dedc3-f6d4-417f-9bca-26052f62191c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" Nov 03 14:31:51.371690 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.370169 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-plugins-config-volume\") pod \"payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm\" (UID: \"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1\") " pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" Nov 03 14:31:51.371690 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.370510 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnhdb\" (UniqueName: \"kubernetes.io/projected/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-kube-api-access-tnhdb\") pod \"payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm\" (UID: \"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1\") " pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" Nov 03 14:31:51.371690 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.371082 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-plugins-config-volume\") pod \"payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm\" (UID: \"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1\") " pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" Nov 03 14:31:51.434846 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.434789 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnhdb\" (UniqueName: \"kubernetes.io/projected/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-kube-api-access-tnhdb\") pod \"payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm\" (UID: \"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1\") " pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" Nov 03 14:31:51.617130 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.617050 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" Nov 03 14:31:51.636985 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.636961 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" Nov 03 14:31:51.935525 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.935498 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr"] Nov 03 14:31:51.936896 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:31:51.936868 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95dedc3_f6d4_417f_9bca_26052f62191c.slice/crio-de4d94c9be40d7d883a85e90d77d3e532ab2eebc21c2f1b329afa1d4b08c719c WatchSource:0}: Error finding container de4d94c9be40d7d883a85e90d77d3e532ab2eebc21c2f1b329afa1d4b08c719c: Status 404 returned error can't find the container with id de4d94c9be40d7d883a85e90d77d3e532ab2eebc21c2f1b329afa1d4b08c719c Nov 03 14:31:51.974508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:51.974482 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm"] Nov 03 14:31:51.991700 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:31:51.991669 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912e73a2_3762_4a7d_8b1e_65ea8fd6a5d1.slice/crio-ec278e7a9ad70a946651862e480a9e7dd0f2f54058cd7d91035bc553398d5725 WatchSource:0}: Error finding container ec278e7a9ad70a946651862e480a9e7dd0f2f54058cd7d91035bc553398d5725: Status 404 returned error can't find the container with id ec278e7a9ad70a946651862e480a9e7dd0f2f54058cd7d91035bc553398d5725 Nov 03 14:31:52.439012 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:52.438972 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" event={"ID":"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1","Type":"ContainerStarted","Data":"c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1"} Nov 03 14:31:52.439456 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:52.439038 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" Nov 03 14:31:52.439456 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:52.439060 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" event={"ID":"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1","Type":"ContainerStarted","Data":"ec278e7a9ad70a946651862e480a9e7dd0f2f54058cd7d91035bc553398d5725"} Nov 03 14:31:52.440632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:52.440585 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" event={"ID":"b95dedc3-f6d4-417f-9bca-26052f62191c","Type":"ContainerStarted","Data":"0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457"} Nov 03 14:31:52.440744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:52.440657 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" event={"ID":"b95dedc3-f6d4-417f-9bca-26052f62191c","Type":"ContainerStarted","Data":"de4d94c9be40d7d883a85e90d77d3e532ab2eebc21c2f1b329afa1d4b08c719c"} Nov 03 14:31:52.440744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:52.440684 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" Nov 03 14:31:52.482186 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:52.482132 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" podStartSLOduration=1.482115672 podStartE2EDuration="1.482115672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:52.481450423 +0000 UTC m=+1188.040086907" watchObservedRunningTime="2026-11-03 14:31:52.482115672 +0000 UTC m=+1188.040752157" Nov 03 14:31:52.524387 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:52.524334 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" podStartSLOduration=2.52431259 podStartE2EDuration="2.52431259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:52.523592494 +0000 UTC m=+1188.082228983" watchObservedRunningTime="2026-11-03 14:31:52.52431259 +0000 UTC m=+1188.082949126" Nov 03 14:31:52.919987 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:52.919951 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs"] Nov 03 14:31:52.924220 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:52.924198 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" Nov 03 14:31:52.944592 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:52.944566 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs"] Nov 03 14:31:52.952821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:52.952800 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4a67e25f-dockercfg-gt2pt\"" Nov 03 14:31:53.065674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:53.048943 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4a67e25f\"" Nov 03 14:31:53.102632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:53.094854 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31248427-a655-46ed-8578-97b7a207372c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs\" (UID: \"31248427-a655-46ed-8578-97b7a207372c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" Nov 03 14:31:53.102632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:53.095142 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dklz4\" (UniqueName: \"kubernetes.io/projected/31248427-a655-46ed-8578-97b7a207372c-kube-api-access-dklz4\") pod \"payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs\" (UID: \"31248427-a655-46ed-8578-97b7a207372c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" Nov 03 14:31:53.217878 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:53.217795 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31248427-a655-46ed-8578-97b7a207372c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs\" (UID: \"31248427-a655-46ed-8578-97b7a207372c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" Nov 03 14:31:53.218045 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:53.218024 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dklz4\" (UniqueName: \"kubernetes.io/projected/31248427-a655-46ed-8578-97b7a207372c-kube-api-access-dklz4\") pod \"payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs\" (UID: \"31248427-a655-46ed-8578-97b7a207372c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" Nov 03 14:31:53.219269 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:53.219246 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31248427-a655-46ed-8578-97b7a207372c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs\" (UID: \"31248427-a655-46ed-8578-97b7a207372c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" Nov 03 14:31:53.240297 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:53.240266 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dklz4\" (UniqueName: \"kubernetes.io/projected/31248427-a655-46ed-8578-97b7a207372c-kube-api-access-dklz4\") pod \"payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs\" (UID: \"31248427-a655-46ed-8578-97b7a207372c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" Nov 03 14:31:53.538725 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:53.538592 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" Nov 03 14:31:53.841842 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:53.841819 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs"] Nov 03 14:31:53.843294 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:31:53.843270 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31248427_a655_46ed_8578_97b7a207372c.slice/crio-52ed90998db6787d19bf91866a825661770170d9519020044a5a7dc30235737b WatchSource:0}: Error finding container 52ed90998db6787d19bf91866a825661770170d9519020044a5a7dc30235737b: Status 404 returned error can't find the container with id 52ed90998db6787d19bf91866a825661770170d9519020044a5a7dc30235737b Nov 03 14:31:54.454901 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:54.454866 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" event={"ID":"31248427-a655-46ed-8578-97b7a207372c","Type":"ContainerStarted","Data":"a8dddc67070b6fa703d0c52a84f675bd6f8208468231ad061920e247acfcde7f"} Nov 03 14:31:54.455064 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:54.454916 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" event={"ID":"31248427-a655-46ed-8578-97b7a207372c","Type":"ContainerStarted","Data":"52ed90998db6787d19bf91866a825661770170d9519020044a5a7dc30235737b"} Nov 03 14:31:54.455064 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:54.455016 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" Nov 03 14:31:54.497973 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:54.497914 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" podStartSLOduration=2.497898504 podStartE2EDuration="2.497898504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:54.496977518 +0000 UTC m=+1190.055614001" watchObservedRunningTime="2026-11-03 14:31:54.497898504 +0000 UTC m=+1190.056534987" Nov 03 14:31:55.243256 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.243225 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz"] Nov 03 14:31:55.247761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.247740 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.271106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.271078 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz"] Nov 03 14:31:55.279647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.279580 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-6653eb-openshift-default-dockercfg-dsnq8\"" Nov 03 14:31:55.297014 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.296987 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-certs\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.297139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.297124 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istiod-ca-cert\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.297187 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.297173 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-podinfo\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.297272 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.297241 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-socket\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.297376 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.297324 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-envoy\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.297492 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.297387 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-token\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.297492 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.297439 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-credential-socket\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.297492 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.297463 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqzn\" (UniqueName: \"kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-kube-api-access-ptqzn\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.297492 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.297488 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-data\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.408976 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.408931 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-certs\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.438765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.410629 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-certs\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.460648 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.439017 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istiod-ca-cert\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.461009 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.460978 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-podinfo\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.461167 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.461151 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-socket\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.461308 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.461293 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-envoy\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.461516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.461497 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-token\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.461724 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.461702 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-credential-socket\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.461850 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.461835 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqzn\" (UniqueName: \"kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-kube-api-access-ptqzn\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.461975 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.461960 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-data\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.483637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.462534 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-data\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.483879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.465001 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-socket\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.483977 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.465226 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-credential-socket\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.484051 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.440083 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istiod-ca-cert\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.484138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.478145 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-envoy\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.506067 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.505992 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqzn\" (UniqueName: \"kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-kube-api-access-ptqzn\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.506756 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.506079 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-podinfo\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.515920 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.515738 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-token\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-njjtz\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.563131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.562847 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:55.913701 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.907226 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz"] Nov 03 14:31:55.941632 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:31:55.935928 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365dbedd_5d89_4991_ad96_ad20ebcfa1b5.slice/crio-62bc6b25df6ca3ef51a9f42931b73a0046145f8a8771144fb69f29370198c6f9 WatchSource:0}: Error finding container 62bc6b25df6ca3ef51a9f42931b73a0046145f8a8771144fb69f29370198c6f9: Status 404 returned error can't find the container with id 62bc6b25df6ca3ef51a9f42931b73a0046145f8a8771144fb69f29370198c6f9 Nov 03 14:31:55.953002 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.952957 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:31:55.953161 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.953048 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:31:55.953161 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:55.953098 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:31:56.518903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:56.518805 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" event={"ID":"365dbedd-5d89-4991-ad96-ad20ebcfa1b5","Type":"ContainerStarted","Data":"a5f85025f300040fe659b8216e5861351e282cab4c6f2d954a3a1b9b6d009b23"} Nov 03 14:31:56.518903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:56.518873 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" event={"ID":"365dbedd-5d89-4991-ad96-ad20ebcfa1b5","Type":"ContainerStarted","Data":"62bc6b25df6ca3ef51a9f42931b73a0046145f8a8771144fb69f29370198c6f9"} Nov 03 14:31:56.564189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:56.564155 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:56.571464 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:56.571418 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" podStartSLOduration=1.571403026 podStartE2EDuration="1.571403026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:56.568928114 +0000 UTC m=+1192.127564600" watchObservedRunningTime="2026-11-03 14:31:56.571403026 +0000 UTC m=+1192.130039510" Nov 03 14:31:57.569629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:57.569587 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:58.374510 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:31:58.374473 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:31:58.530347 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:58.530318 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:31:58.531350 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:31:58.531326 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:32:01.064990 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.064956 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85"] Nov 03 14:32:01.069506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.069475 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" Nov 03 14:32:01.087955 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.087920 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-plugins-config-volume\") pod \"payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85\" (UID: \"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60\") " pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" Nov 03 14:32:01.088115 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.088096 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8tjv\" (UniqueName: \"kubernetes.io/projected/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-kube-api-access-l8tjv\") pod \"payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85\" (UID: \"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60\") " pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" Nov 03 14:32:01.131075 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.131040 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-6653eb\"" Nov 03 14:32:01.131483 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.131465 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-6653eb-dockercfg-p79hx\"" Nov 03 14:32:01.149629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.145691 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85"] Nov 03 14:32:01.202586 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.202549 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8tjv\" (UniqueName: \"kubernetes.io/projected/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-kube-api-access-l8tjv\") pod \"payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85\" (UID: \"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60\") " pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" Nov 03 14:32:01.212646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.206823 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-plugins-config-volume\") pod \"payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85\" (UID: \"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60\") " pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" Nov 03 14:32:01.235933 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.235887 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-plugins-config-volume\") pod \"payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85\" (UID: \"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60\") " pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" Nov 03 14:32:01.255962 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.255920 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8tjv\" (UniqueName: \"kubernetes.io/projected/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-kube-api-access-l8tjv\") pod \"payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85\" (UID: \"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60\") " pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" Nov 03 14:32:01.385079 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.384985 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" Nov 03 14:32:01.699048 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:01.698955 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85"] Nov 03 14:32:01.720414 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:32:01.720381 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ae7db7_52d6_41fd_b4c3_0cc19e834f60.slice/crio-c42b44e00d91eb385cf0bdd1b12db2dd72a143cea1c8eec4a544d9e9ee0631e3 WatchSource:0}: Error finding container c42b44e00d91eb385cf0bdd1b12db2dd72a143cea1c8eec4a544d9e9ee0631e3: Status 404 returned error can't find the container with id c42b44e00d91eb385cf0bdd1b12db2dd72a143cea1c8eec4a544d9e9ee0631e3 Nov 03 14:32:02.555909 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:02.555862 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" event={"ID":"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60","Type":"ContainerStarted","Data":"1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1"} Nov 03 14:32:02.556389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:02.555957 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" event={"ID":"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60","Type":"ContainerStarted","Data":"c42b44e00d91eb385cf0bdd1b12db2dd72a143cea1c8eec4a544d9e9ee0631e3"} Nov 03 14:32:02.556389 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:02.556098 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" Nov 03 14:32:02.605848 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:02.605788 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" podStartSLOduration=1.6057701149999999 podStartE2EDuration="1.605770115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:02.603939827 +0000 UTC m=+1198.162576313" watchObservedRunningTime="2026-11-03 14:32:02.605770115 +0000 UTC m=+1198.164406600" Nov 03 14:32:03.457631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:03.456775 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" Nov 03 14:32:03.461633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:03.457987 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" Nov 03 14:32:05.512561 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:05.512533 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" Nov 03 14:32:13.607733 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:13.607698 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" Nov 03 14:32:17.281547 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:17.281515 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f"] Nov 03 14:32:22.282280 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.282224 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" podUID="0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e" containerName="payload-processing" containerID="cri-o://18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d" gracePeriod=25 Nov 03 14:32:22.592864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.592838 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:32:22.707353 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.707324 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-routing-overlay-volume\") pod \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " Nov 03 14:32:22.707535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.707398 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-provider-credentials\") pod \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " Nov 03 14:32:22.707535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.707529 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-plugins-config-volume\") pod \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " Nov 03 14:32:22.707686 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.707579 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-kube-api-access-6vrrl\" (UniqueName: \"kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-kube-api-access-6vrrl\") pod \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\" (UID: \"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e\") " Nov 03 14:32:22.707750 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.707686 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-routing-overlay-volume" pod "0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e" (UID: "0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:32:22.707981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.707955 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-plugins-config-volume" pod "0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e" (UID: "0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:32:22.708332 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.708313 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:22.708485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.708460 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:22.709876 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.709851 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-provider-credentials" pod "0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e" (UID: "0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:22.709953 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.709904 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-kube-api-access-6vrrl" pod "0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e" (UID: "0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e"). InnerVolumeSpecName "kube-api-access-6vrrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:22.818809 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.814768 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vrrl\" (UniqueName: \"kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-kube-api-access-6vrrl\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:22.818809 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:22.814804 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:23.199594 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:23.199555 6389 generic.go:356] "Generic (PLEG): container finished" podID="0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e" containerID="18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d" exitCode=0 Nov 03 14:32:23.199920 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:23.199631 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" Nov 03 14:32:23.199920 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:23.199645 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" event={"ID":"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e","Type":"ContainerDied","Data":"18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d"} Nov 03 14:32:23.199920 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:23.199736 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f" event={"ID":"0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e","Type":"ContainerDied","Data":"8e4798a7cac3c2fcaf62e679c2021fbb4e8283b8ec3446350bed9a791d938113"} Nov 03 14:32:23.199920 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:23.199769 6389 scope.go:122] "RemoveContainer" containerID="18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d" Nov 03 14:32:23.210114 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:23.210098 6389 scope.go:122] "RemoveContainer" containerID="18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d" Nov 03 14:32:23.210367 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:32:23.210350 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d\": container with ID starting with 18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d not found: ID does not exist" containerID="18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d" Nov 03 14:32:23.210432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:23.210379 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d"} err="failed to get container status \"18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d\": rpc error: code = NotFound desc = could not find container \"18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d\": container with ID starting with 18c52830c89f85c6811288572438323c49f30804659acc335102d83919dbf20d not found: ID does not exist" Nov 03 14:32:23.257075 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:23.257038 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f"] Nov 03 14:32:23.269517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:23.269488 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-54b5cdz5f"] Nov 03 14:32:24.666455 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:24.666423 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e" path="/var/lib/kubelet/pods/0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e/volumes" Nov 03 14:32:28.237813 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:28.237778 6389 generic.go:356] "Generic (PLEG): container finished" podID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerID="37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8" exitCode=0 Nov 03 14:32:28.238237 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:28.237850 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" event={"ID":"3816b61f-c6dc-4b2f-9bb4-f679aa95285f","Type":"ContainerDied","Data":"37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8"} Nov 03 14:32:28.238288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:28.238274 6389 scope.go:122] "RemoveContainer" containerID="37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8" Nov 03 14:32:28.239744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:28.239720 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:32:28.239805 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:28.239791 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:32:28.239839 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:28.239826 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:32:28.382834 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:32:28.382794 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:32:29.245425 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:29.245391 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" event={"ID":"3816b61f-c6dc-4b2f-9bb4-f679aa95285f","Type":"ContainerStarted","Data":"e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71"} Nov 03 14:32:29.482395 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:29.482308 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:32:29.482619 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:29.482543 6389 patch_prober.go:29] interesting pod/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:32:29.482709 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:29.482598 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 14:32:30.482109 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:30.482077 6389 patch_prober.go:29] interesting pod/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:32:30.482525 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:30.482127 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 14:32:31.481585 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:31.481552 6389 patch_prober.go:29] interesting pod/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:32:31.481785 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:31.481658 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 14:32:32.490522 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:32.490487 6389 patch_prober.go:29] interesting pod/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:32:32.492489 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:32.491156 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 14:32:33.481507 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:33.481466 6389 patch_prober.go:29] interesting pod/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:32:33.481709 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:33.481543 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 14:32:34.391911 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:34.391872 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:32:34.482169 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:34.482137 6389 patch_prober.go:29] interesting pod/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:32:34.482385 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:34.482211 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 14:32:35.697040 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:35.696996 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:32:35.700878 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:35.700851 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:32:42.351009 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:42.350977 6389 generic.go:356] "Generic (PLEG): container finished" podID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerID="a5f85025f300040fe659b8216e5861351e282cab4c6f2d954a3a1b9b6d009b23" exitCode=0 Nov 03 14:32:42.351582 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:42.351057 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" event={"ID":"365dbedd-5d89-4991-ad96-ad20ebcfa1b5","Type":"ContainerDied","Data":"a5f85025f300040fe659b8216e5861351e282cab4c6f2d954a3a1b9b6d009b23"} Nov 03 14:32:42.351698 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:42.351682 6389 scope.go:122] "RemoveContainer" containerID="a5f85025f300040fe659b8216e5861351e282cab4c6f2d954a3a1b9b6d009b23" Nov 03 14:32:42.353979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:42.353941 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:32:42.354188 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:42.354165 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:32:42.354285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:42.354219 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:32:43.361083 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:43.361051 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" event={"ID":"365dbedd-5d89-4991-ad96-ad20ebcfa1b5","Type":"ContainerStarted","Data":"94959755c4be219a077ebe9e201a52ffcdeb5b9d9f0220db4a8a97a209dab966"} Nov 03 14:32:43.543832 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:43.543796 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:32:43.563654 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:43.563621 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:32:43.564968 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:43.564940 6389 patch_prober.go:29] interesting pod/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 14:32:43.565127 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:43.565005 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 14:32:44.564249 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:44.564212 6389 patch_prober.go:29] interesting pod/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 14:32:44.564803 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:44.564261 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 14:32:45.563832 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:45.563798 6389 patch_prober.go:29] interesting pod/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 14:32:45.564051 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:45.563857 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 14:32:46.550107 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.550065 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm"] Nov 03 14:32:46.564585 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.564555 6389 patch_prober.go:29] interesting pod/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 14:32:46.564778 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.564644 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 14:32:46.594262 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.594218 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr"] Nov 03 14:32:46.666793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.666532 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt"] Nov 03 14:32:46.668143 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.668121 6389 cpu_manager.go:414] "containerMap: removing container" podUID="0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e" containerName="payload-processing" Nov 03 14:32:46.668338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.668323 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e" containerName="payload-processing" Nov 03 14:32:46.668650 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.668637 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4de2e2-0f1b-49b7-b9db-46a1e3651f4e" containerName="payload-processing" Nov 03 14:32:46.682077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.682052 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.690902 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.690868 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt"] Nov 03 14:32:46.699478 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.699453 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-1ffaa0-openshift-default-dockercfg-k7w6w\"" Nov 03 14:32:46.713047 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.712027 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.713047 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.712086 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.713047 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.712107 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.713047 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.712124 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.713047 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.712170 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.713047 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.712200 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.713047 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.712240 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.713047 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.712260 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cddad1-9fbc-482a-9347-b8f02643e2ff-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.713047 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.712292 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqtcp\" (UniqueName: \"kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-kube-api-access-kqtcp\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.813923 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.813833 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.813923 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.813877 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.813923 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.813896 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.814220 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.813957 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.814220 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.813994 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.814220 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.814054 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.814220 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.814071 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cddad1-9fbc-482a-9347-b8f02643e2ff-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.814220 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.814104 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqtcp\" (UniqueName: \"kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-kube-api-access-kqtcp\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.814220 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.814206 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.814537 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.814236 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.814655 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.814538 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.814872 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.814830 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.814953 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.814896 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.815797 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.815773 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cddad1-9fbc-482a-9347-b8f02643e2ff-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.817227 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.817203 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.817466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.817446 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.907040 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.907010 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqtcp\" (UniqueName: \"kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-kube-api-access-kqtcp\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:46.908765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:46.908743 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:47.005806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.005777 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:47.382850 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.382814 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt"] Nov 03 14:32:47.473677 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.471384 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:32:47.473677 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.471478 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:32:47.473677 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.471520 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:32:47.569629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.565000 6389 patch_prober.go:29] interesting pod/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 14:32:47.569629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.565066 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 14:32:47.596823 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.595943 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92"] Nov 03 14:32:47.607819 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.607790 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.623673 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.614340 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92"] Nov 03 14:32:47.637366 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.635358 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-6653eb\"/\"autores-model-6653eb-kserve-self-signed-certs\"" Nov 03 14:32:47.637366 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.636135 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-6653eb\"/\"openshift-service-ca.crt\"" Nov 03 14:32:47.637366 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.636230 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-6653eb\"/\"kube-root-ca.crt\"" Nov 03 14:32:47.638171 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.637800 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-6653eb\"/\"default-dockercfg-9dkj7\"" Nov 03 14:32:47.662430 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.662396 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tmp-dir\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.662668 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.662501 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-dshm\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.662668 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.662568 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-model-cache\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.662810 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.662713 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tls-certs\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.662871 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.662812 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdntv\" (UniqueName: \"kubernetes.io/projected/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-kube-api-access-gdntv\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.663845 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.663794 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-home\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.765650 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.765589 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-home\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.765899 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.765877 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tmp-dir\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.766053 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.765930 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-home\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.766144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.766122 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tmp-dir\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.766241 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.766229 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-dshm\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.766350 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.766340 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-model-cache\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.766443 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.766433 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tls-certs\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.766572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.766557 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdntv\" (UniqueName: \"kubernetes.io/projected/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-kube-api-access-gdntv\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.766743 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.766721 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-model-cache\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.768546 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.768525 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-dshm\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.769303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.769281 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tls-certs\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.791143 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.791113 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdntv\" (UniqueName: \"kubernetes.io/projected/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-kube-api-access-gdntv\") pod \"autores-model-6653eb-kserve-587557877c-zqh92\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:47.967642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:47.966150 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:48.389200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:48.389156 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92"] Nov 03 14:32:48.390649 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:32:48.390570 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4b26ed3_e0d9_4fe1_83d4_ad1c47ee65b6.slice/crio-06dfd027d6e6ae0d1759bb17281b68934c4ed0ae5212dd94f648d86add3d0b81 WatchSource:0}: Error finding container 06dfd027d6e6ae0d1759bb17281b68934c4ed0ae5212dd94f648d86add3d0b81: Status 404 returned error can't find the container with id 06dfd027d6e6ae0d1759bb17281b68934c4ed0ae5212dd94f648d86add3d0b81 Nov 03 14:32:48.443679 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:48.443637 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" event={"ID":"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6","Type":"ContainerStarted","Data":"06dfd027d6e6ae0d1759bb17281b68934c4ed0ae5212dd94f648d86add3d0b81"} Nov 03 14:32:48.445421 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:48.445392 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" event={"ID":"86cddad1-9fbc-482a-9347-b8f02643e2ff","Type":"ContainerStarted","Data":"3defcf887dd72b1c4a1bc84198db677b3c9f8cc923db275220462e8e38942686"} Nov 03 14:32:48.445554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:48.445439 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" event={"ID":"86cddad1-9fbc-482a-9347-b8f02643e2ff","Type":"ContainerStarted","Data":"785a94d610d87fef23100a50206311cc1a4dd984a099575950360484b670c2ae"} Nov 03 14:32:48.498061 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:48.497989 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" podStartSLOduration=2.497961406 podStartE2EDuration="2.497961406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:48.493192885 +0000 UTC m=+1244.051829370" watchObservedRunningTime="2026-11-03 14:32:48.497961406 +0000 UTC m=+1244.056597892" Nov 03 14:32:48.564196 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:48.564150 6389 patch_prober.go:29] interesting pod/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 14:32:48.564344 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:48.564222 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 14:32:49.006959 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:49.006926 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:49.008764 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:49.008740 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 14:32:49.008901 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:49.008787 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" podUID="86cddad1-9fbc-482a-9347-b8f02643e2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 14:32:49.453516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:49.453475 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" event={"ID":"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6","Type":"ContainerStarted","Data":"a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a"} Nov 03 14:32:49.501557 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:49.501506 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" podStartSLOduration=2.5014932930000002 podStartE2EDuration="2.501493293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:49.500533357 +0000 UTC m=+1245.059169842" watchObservedRunningTime="2026-11-03 14:32:49.501493293 +0000 UTC m=+1245.060129777" Nov 03 14:32:49.568702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:49.568672 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:32:49.569739 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:49.569716 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:32:49.973484 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:49.973447 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:50.008640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:50.006230 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 14:32:50.008640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:50.006292 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" podUID="86cddad1-9fbc-482a-9347-b8f02643e2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 14:32:51.006833 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:51.006792 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 14:32:51.007029 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:51.006848 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" podUID="86cddad1-9fbc-482a-9347-b8f02643e2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 14:32:51.551425 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:51.551358 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" podUID="912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1" containerName="payload-processing" containerID="cri-o://c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1" gracePeriod=25 Nov 03 14:32:51.594914 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:51.594853 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" podUID="b95dedc3-f6d4-417f-9bca-26052f62191c" containerName="payload-pre-processing" containerID="cri-o://0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457" gracePeriod=25 Nov 03 14:32:51.985252 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:32:51.980202 6389 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95dedc3_f6d4_417f_9bca_26052f62191c.slice/crio-0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95dedc3_f6d4_417f_9bca_26052f62191c.slice/crio-conmon-0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457.scope\": RecentStats: unable to find data in memory cache]" Nov 03 14:32:52.140694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.137162 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 14:32:52.169882 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.169468 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" podUID="86cddad1-9fbc-482a-9347-b8f02643e2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 14:32:52.387063 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.387035 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" Nov 03 14:32:52.390684 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.390663 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" Nov 03 14:32:52.484097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.483804 6389 generic.go:356] "Generic (PLEG): container finished" podID="912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1" containerID="c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1" exitCode=0 Nov 03 14:32:52.484097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.483942 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" Nov 03 14:32:52.484378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.484140 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" event={"ID":"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1","Type":"ContainerDied","Data":"c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1"} Nov 03 14:32:52.484378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.484193 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm" event={"ID":"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1","Type":"ContainerDied","Data":"ec278e7a9ad70a946651862e480a9e7dd0f2f54058cd7d91035bc553398d5725"} Nov 03 14:32:52.484378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.484223 6389 scope.go:122] "RemoveContainer" containerID="c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1" Nov 03 14:32:52.488104 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.488078 6389 generic.go:356] "Generic (PLEG): container finished" podID="b95dedc3-f6d4-417f-9bca-26052f62191c" containerID="0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457" exitCode=0 Nov 03 14:32:52.488224 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.488127 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" event={"ID":"b95dedc3-f6d4-417f-9bca-26052f62191c","Type":"ContainerDied","Data":"0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457"} Nov 03 14:32:52.488224 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.488167 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" event={"ID":"b95dedc3-f6d4-417f-9bca-26052f62191c","Type":"ContainerDied","Data":"de4d94c9be40d7d883a85e90d77d3e532ab2eebc21c2f1b329afa1d4b08c719c"} Nov 03 14:32:52.488349 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.488323 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr" Nov 03 14:32:52.507491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.507467 6389 scope.go:122] "RemoveContainer" containerID="c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1" Nov 03 14:32:52.508161 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:32:52.508134 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1\": container with ID starting with c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1 not found: ID does not exist" containerID="c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1" Nov 03 14:32:52.508274 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.508172 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1"} err="failed to get container status \"c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1\": rpc error: code = NotFound desc = could not find container \"c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1\": container with ID starting with c00958c00af5441e1840c2da7925150dc94ee2960d25df3aa6559370f045bcc1 not found: ID does not exist" Nov 03 14:32:52.508274 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.508191 6389 scope.go:122] "RemoveContainer" containerID="0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457" Nov 03 14:32:52.558857 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.558819 6389 scope.go:122] "RemoveContainer" containerID="0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457" Nov 03 14:32:52.559701 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:32:52.559684 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457\": container with ID starting with 0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457 not found: ID does not exist" containerID="0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457" Nov 03 14:32:52.559841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.559822 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457"} err="failed to get container status \"0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457\": rpc error: code = NotFound desc = could not find container \"0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457\": container with ID starting with 0b1351fec0987ef73eb7e9240e2640d4b5d65203bbcdab3975f504d8cef5f457 not found: ID does not exist" Nov 03 14:32:52.568486 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.568457 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95dedc3-f6d4-417f-9bca-26052f62191c-kube-api-access-8zcm6\" (UniqueName: \"kubernetes.io/projected/b95dedc3-f6d4-417f-9bca-26052f62191c-kube-api-access-8zcm6\") pod \"b95dedc3-f6d4-417f-9bca-26052f62191c\" (UID: \"b95dedc3-f6d4-417f-9bca-26052f62191c\") " Nov 03 14:32:52.568840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.568822 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b95dedc3-f6d4-417f-9bca-26052f62191c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95dedc3-f6d4-417f-9bca-26052f62191c-plugins-config-volume\") pod \"b95dedc3-f6d4-417f-9bca-26052f62191c\" (UID: \"b95dedc3-f6d4-417f-9bca-26052f62191c\") " Nov 03 14:32:52.569039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.569009 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-kube-api-access-tnhdb\" (UniqueName: \"kubernetes.io/projected/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-kube-api-access-tnhdb\") pod \"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1\" (UID: \"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1\") " Nov 03 14:32:52.569203 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.569182 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-plugins-config-volume\") pod \"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1\" (UID: \"912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1\") " Nov 03 14:32:52.569722 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.569571 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95dedc3-f6d4-417f-9bca-26052f62191c-plugins-config-volume" pod "b95dedc3-f6d4-417f-9bca-26052f62191c" (UID: "b95dedc3-f6d4-417f-9bca-26052f62191c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:32:52.570160 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.570143 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95dedc3-f6d4-417f-9bca-26052f62191c-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.570867 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.570847 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-plugins-config-volume" pod "912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1" (UID: "912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:32:52.578905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.578867 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95dedc3-f6d4-417f-9bca-26052f62191c-kube-api-access-8zcm6" pod "b95dedc3-f6d4-417f-9bca-26052f62191c" (UID: "b95dedc3-f6d4-417f-9bca-26052f62191c"). InnerVolumeSpecName "kube-api-access-8zcm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:52.579956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.579906 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-kube-api-access-tnhdb" pod "912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1" (UID: "912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1"). InnerVolumeSpecName "kube-api-access-tnhdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:52.722601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.705836 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnhdb\" (UniqueName: \"kubernetes.io/projected/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-kube-api-access-tnhdb\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.722601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.705869 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.722601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.705883 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zcm6\" (UniqueName: \"kubernetes.io/projected/b95dedc3-f6d4-417f-9bca-26052f62191c-kube-api-access-8zcm6\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.846816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.846753 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm"] Nov 03 14:32:52.850257 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.850227 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-09d577dc-86ccb7d997-6d2sm"] Nov 03 14:32:52.887336 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.887302 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr"] Nov 03 14:32:52.890133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:52.890109 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-09d577dc-5cc6bb6659-5g2dr"] Nov 03 14:32:53.006384 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:53.006344 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 14:32:53.006547 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:53.006410 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" podUID="86cddad1-9fbc-482a-9347-b8f02643e2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 14:32:53.149530 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:53.149424 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw"] Nov 03 14:32:53.150071 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:53.150043 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" containerID="cri-o://e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71" gracePeriod=30 Nov 03 14:32:54.007098 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:54.007063 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 14:32:54.007529 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:54.007115 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" podUID="86cddad1-9fbc-482a-9347-b8f02643e2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 14:32:54.620701 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:54.620666 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1" path="/var/lib/kubelet/pods/912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1/volumes" Nov 03 14:32:54.621254 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:54.621229 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b95dedc3-f6d4-417f-9bca-26052f62191c" path="/var/lib/kubelet/pods/b95dedc3-f6d4-417f-9bca-26052f62191c/volumes" Nov 03 14:32:55.691137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:55.691107 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:55.691706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:55.691534 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:55.692391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:55.692364 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:32:57.968782 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:57.968744 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:57.981768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:57.981699 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:58.381839 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:32:58.381799 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:32:58.421868 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.421844 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:32:58.548107 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.548028 6389 generic.go:356] "Generic (PLEG): container finished" podID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerID="e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71" exitCode=0 Nov 03 14:32:58.548260 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.548135 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" Nov 03 14:32:58.548260 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.548159 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" event={"ID":"3816b61f-c6dc-4b2f-9bb4-f679aa95285f","Type":"ContainerDied","Data":"e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71"} Nov 03 14:32:58.548260 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.548211 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw" event={"ID":"3816b61f-c6dc-4b2f-9bb4-f679aa95285f","Type":"ContainerDied","Data":"a03c485748f04f33abde21d5805d28c5f0336970235760c3cc3e121ed7199755"} Nov 03 14:32:58.548260 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.548233 6389 scope.go:122] "RemoveContainer" containerID="e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71" Nov 03 14:32:58.558585 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.558568 6389 scope.go:122] "RemoveContainer" containerID="37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8" Nov 03 14:32:58.559956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.559939 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:32:58.567577 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.567562 6389 scope.go:122] "RemoveContainer" containerID="e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71" Nov 03 14:32:58.567831 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:32:58.567814 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71\": container with ID starting with e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71 not found: ID does not exist" containerID="e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71" Nov 03 14:32:58.567891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.567838 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71"} err="failed to get container status \"e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71\": rpc error: code = NotFound desc = could not find container \"e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71\": container with ID starting with e24d09c013bb4f22af65fc9fa587b456773879fb5360de0436d99c76cb0dfd71 not found: ID does not exist" Nov 03 14:32:58.567891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.567853 6389 scope.go:122] "RemoveContainer" containerID="37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8" Nov 03 14:32:58.568083 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:32:58.568065 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8\": container with ID starting with 37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8 not found: ID does not exist" containerID="37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8" Nov 03 14:32:58.568128 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.568090 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8"} err="failed to get container status \"37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8\": rpc error: code = NotFound desc = could not find container \"37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8\": container with ID starting with 37a2f27f0dbc7ff3f92cdd4c0e23f9d4684f7836a10118e0fd23c3276ae587a8 not found: ID does not exist" Nov 03 14:32:58.582363 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.582335 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-socket\") pod \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " Nov 03 14:32:58.582460 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.582390 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istiod-ca-cert\") pod \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " Nov 03 14:32:58.582460 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.582430 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-kube-api-access-sszsb\" (UniqueName: \"kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-kube-api-access-sszsb\") pod \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " Nov 03 14:32:58.582561 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.582469 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-envoy\") pod \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " Nov 03 14:32:58.582561 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.582507 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-token\" (UniqueName: \"kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-token\") pod \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " Nov 03 14:32:58.582704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.582577 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-credential-socket\") pod \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " Nov 03 14:32:58.582704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.582639 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-data\") pod \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " Nov 03 14:32:58.582704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.582641 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-socket" pod "3816b61f-c6dc-4b2f-9bb4-f679aa95285f" (UID: "3816b61f-c6dc-4b2f-9bb4-f679aa95285f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:58.582704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.582672 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-certs\") pod \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " Nov 03 14:32:58.582914 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.582725 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-podinfo\") pod \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\" (UID: \"3816b61f-c6dc-4b2f-9bb4-f679aa95285f\") " Nov 03 14:32:58.582914 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.582877 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-credential-socket" pod "3816b61f-c6dc-4b2f-9bb4-f679aa95285f" (UID: "3816b61f-c6dc-4b2f-9bb4-f679aa95285f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:58.582914 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.582898 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-data" pod "3816b61f-c6dc-4b2f-9bb4-f679aa95285f" (UID: "3816b61f-c6dc-4b2f-9bb4-f679aa95285f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:58.583081 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.583058 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-certs" pod "3816b61f-c6dc-4b2f-9bb4-f679aa95285f" (UID: "3816b61f-c6dc-4b2f-9bb4-f679aa95285f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:58.583252 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.583129 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istiod-ca-cert" pod "3816b61f-c6dc-4b2f-9bb4-f679aa95285f" (UID: "3816b61f-c6dc-4b2f-9bb4-f679aa95285f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:32:58.583351 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.583335 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:58.583414 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.583357 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:58.583414 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.583371 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:58.583414 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.583384 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:58.583414 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.583396 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:58.584822 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.584797 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-kube-api-access-sszsb" pod "3816b61f-c6dc-4b2f-9bb4-f679aa95285f" (UID: "3816b61f-c6dc-4b2f-9bb4-f679aa95285f"). InnerVolumeSpecName "kube-api-access-sszsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:58.585032 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.585008 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-token" pod "3816b61f-c6dc-4b2f-9bb4-f679aa95285f" (UID: "3816b61f-c6dc-4b2f-9bb4-f679aa95285f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:58.585114 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.585075 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-envoy" pod "3816b61f-c6dc-4b2f-9bb4-f679aa95285f" (UID: "3816b61f-c6dc-4b2f-9bb4-f679aa95285f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:58.585289 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.585276 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-podinfo" pod "3816b61f-c6dc-4b2f-9bb4-f679aa95285f" (UID: "3816b61f-c6dc-4b2f-9bb4-f679aa95285f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:32:58.685020 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.684991 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:58.685020 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.685018 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sszsb\" (UniqueName: \"kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-kube-api-access-sszsb\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:58.685020 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.685033 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:58.685258 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.685044 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3816b61f-c6dc-4b2f-9bb4-f679aa95285f-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:32:58.893766 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.893690 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw"] Nov 03 14:32:58.906775 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:58.906748 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-09d577dc-openshift-default-6cfd85bd75-8zwlw"] Nov 03 14:32:59.641314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.641277 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6"] Nov 03 14:32:59.642091 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.642070 6389 cpu_manager.go:414] "containerMap: removing container" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" Nov 03 14:32:59.642199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.642094 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" Nov 03 14:32:59.642199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.642120 6389 cpu_manager.go:414] "containerMap: removing container" podUID="912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1" containerName="payload-processing" Nov 03 14:32:59.642199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.642128 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1" containerName="payload-processing" Nov 03 14:32:59.642199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.642144 6389 cpu_manager.go:414] "containerMap: removing container" podUID="b95dedc3-f6d4-417f-9bca-26052f62191c" containerName="payload-pre-processing" Nov 03 14:32:59.642199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.642152 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b95dedc3-f6d4-417f-9bca-26052f62191c" containerName="payload-pre-processing" Nov 03 14:32:59.642199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.642166 6389 cpu_manager.go:414] "containerMap: removing container" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" Nov 03 14:32:59.642199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.642173 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" Nov 03 14:32:59.642529 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.642357 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" Nov 03 14:32:59.642529 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.642373 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="912e73a2-3762-4a7d-8b1e-65ea8fd6a5d1" containerName="payload-processing" Nov 03 14:32:59.642529 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.642384 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="b95dedc3-f6d4-417f-9bca-26052f62191c" containerName="payload-pre-processing" Nov 03 14:32:59.647637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.647579 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" Nov 03 14:32:59.664945 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.664920 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-95af53a1-dockercfg-5qrm2\"" Nov 03 14:32:59.665062 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.664991 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-95af53a1\"" Nov 03 14:32:59.667206 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.667184 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6"] Nov 03 14:32:59.836634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.834078 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql"] Nov 03 14:32:59.841635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.840739 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" containerName="istio-proxy" Nov 03 14:32:59.874106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.864585 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3676abfd-9064-436e-83be-4aa79bf35f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6\" (UID: \"3676abfd-9064-436e-83be-4aa79bf35f01\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" Nov 03 14:32:59.874106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.864729 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvgcz\" (UniqueName: \"kubernetes.io/projected/3676abfd-9064-436e-83be-4aa79bf35f01-kube-api-access-nvgcz\") pod \"payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6\" (UID: \"3676abfd-9064-436e-83be-4aa79bf35f01\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" Nov 03 14:32:59.934051 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.934015 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql"] Nov 03 14:32:59.934380 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.934363 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" Nov 03 14:32:59.979400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.979368 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3676abfd-9064-436e-83be-4aa79bf35f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6\" (UID: \"3676abfd-9064-436e-83be-4aa79bf35f01\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" Nov 03 14:32:59.979600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.979430 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjfxw\" (UniqueName: \"kubernetes.io/projected/07b326a2-b5e6-4030-b9bd-4d85d19107c3-kube-api-access-hjfxw\") pod \"payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql\" (UID: \"07b326a2-b5e6-4030-b9bd-4d85d19107c3\") " pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" Nov 03 14:32:59.979600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.979465 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b326a2-b5e6-4030-b9bd-4d85d19107c3-plugins-config-volume\") pod \"payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql\" (UID: \"07b326a2-b5e6-4030-b9bd-4d85d19107c3\") " pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" Nov 03 14:32:59.979600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.979586 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvgcz\" (UniqueName: \"kubernetes.io/projected/3676abfd-9064-436e-83be-4aa79bf35f01-kube-api-access-nvgcz\") pod \"payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6\" (UID: \"3676abfd-9064-436e-83be-4aa79bf35f01\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" Nov 03 14:32:59.980070 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:32:59.980050 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3676abfd-9064-436e-83be-4aa79bf35f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6\" (UID: \"3676abfd-9064-436e-83be-4aa79bf35f01\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" Nov 03 14:33:00.001003 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:00.000968 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvgcz\" (UniqueName: \"kubernetes.io/projected/3676abfd-9064-436e-83be-4aa79bf35f01-kube-api-access-nvgcz\") pod \"payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6\" (UID: \"3676abfd-9064-436e-83be-4aa79bf35f01\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" Nov 03 14:33:00.081573 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:00.081535 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjfxw\" (UniqueName: \"kubernetes.io/projected/07b326a2-b5e6-4030-b9bd-4d85d19107c3-kube-api-access-hjfxw\") pod \"payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql\" (UID: \"07b326a2-b5e6-4030-b9bd-4d85d19107c3\") " pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" Nov 03 14:33:00.081573 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:00.081579 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b326a2-b5e6-4030-b9bd-4d85d19107c3-plugins-config-volume\") pod \"payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql\" (UID: \"07b326a2-b5e6-4030-b9bd-4d85d19107c3\") " pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" Nov 03 14:33:00.082305 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:00.082285 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b326a2-b5e6-4030-b9bd-4d85d19107c3-plugins-config-volume\") pod \"payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql\" (UID: \"07b326a2-b5e6-4030-b9bd-4d85d19107c3\") " pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" Nov 03 14:33:00.103992 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:00.103958 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjfxw\" (UniqueName: \"kubernetes.io/projected/07b326a2-b5e6-4030-b9bd-4d85d19107c3-kube-api-access-hjfxw\") pod \"payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql\" (UID: \"07b326a2-b5e6-4030-b9bd-4d85d19107c3\") " pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" Nov 03 14:33:00.252230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:00.252151 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" Nov 03 14:33:00.262680 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:00.262652 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" Nov 03 14:33:00.560864 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:33:00.560832 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b326a2_b5e6_4030_b9bd_4d85d19107c3.slice/crio-b79aa3ab6477141cbadd226745e4e1e0df54936a9763318ead6ee43c177d6461 WatchSource:0}: Error finding container b79aa3ab6477141cbadd226745e4e1e0df54936a9763318ead6ee43c177d6461: Status 404 returned error can't find the container with id b79aa3ab6477141cbadd226745e4e1e0df54936a9763318ead6ee43c177d6461 Nov 03 14:33:00.561119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:00.561094 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql"] Nov 03 14:33:00.572976 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:00.572954 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6"] Nov 03 14:33:00.574811 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:33:00.574788 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3676abfd_9064_436e_83be_4aa79bf35f01.slice/crio-9cf7f0fe5f177694aa33e36b31fe09fd30262ae27997ea9f669784bc06c5ba58 WatchSource:0}: Error finding container 9cf7f0fe5f177694aa33e36b31fe09fd30262ae27997ea9f669784bc06c5ba58: Status 404 returned error can't find the container with id 9cf7f0fe5f177694aa33e36b31fe09fd30262ae27997ea9f669784bc06c5ba58 Nov 03 14:33:00.620771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:00.620747 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3816b61f-c6dc-4b2f-9bb4-f679aa95285f" path="/var/lib/kubelet/pods/3816b61f-c6dc-4b2f-9bb4-f679aa95285f/volumes" Nov 03 14:33:01.571503 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:01.571467 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" event={"ID":"07b326a2-b5e6-4030-b9bd-4d85d19107c3","Type":"ContainerStarted","Data":"51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274"} Nov 03 14:33:01.571961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:01.571532 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" event={"ID":"07b326a2-b5e6-4030-b9bd-4d85d19107c3","Type":"ContainerStarted","Data":"b79aa3ab6477141cbadd226745e4e1e0df54936a9763318ead6ee43c177d6461"} Nov 03 14:33:01.571961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:01.571565 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" Nov 03 14:33:01.573308 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:01.573283 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" event={"ID":"3676abfd-9064-436e-83be-4aa79bf35f01","Type":"ContainerStarted","Data":"d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70"} Nov 03 14:33:01.573425 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:01.573324 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" event={"ID":"3676abfd-9064-436e-83be-4aa79bf35f01","Type":"ContainerStarted","Data":"9cf7f0fe5f177694aa33e36b31fe09fd30262ae27997ea9f669784bc06c5ba58"} Nov 03 14:33:01.573425 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:01.573364 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" Nov 03 14:33:01.627870 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:01.627821 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" podStartSLOduration=2.627806676 podStartE2EDuration="2.627806676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:01.614262546 +0000 UTC m=+1257.172899029" watchObservedRunningTime="2026-11-03 14:33:01.627806676 +0000 UTC m=+1257.186443195" Nov 03 14:33:01.656409 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:01.656333 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" podStartSLOduration=2.656315395 podStartE2EDuration="2.656315395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:01.655989479 +0000 UTC m=+1257.214625964" watchObservedRunningTime="2026-11-03 14:33:01.656315395 +0000 UTC m=+1257.214951885" Nov 03 14:33:04.555951 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.555902 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn"] Nov 03 14:33:04.628636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.562727 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:04.628636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.608424 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn"] Nov 03 14:33:04.628636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.611868 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"routing-overlay\"" Nov 03 14:33:04.628636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.611946 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"openshift-service-ca.crt\"" Nov 03 14:33:04.628636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.612027 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b4361f\"" Nov 03 14:33:04.628636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.612087 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-95ac97e0ca1e\"" Nov 03 14:33:04.628636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.612153 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-c88df00e610c\"" Nov 03 14:33:04.642631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.639325 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-e2e--29c79c46-dockercfg-xvjqk\"" Nov 03 14:33:04.642631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.639418 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"kube-root-ca.crt\"" Nov 03 14:33:04.704633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.701575 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:04.704633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.701763 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:04.704633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.701822 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:04.704633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.701948 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phjvt\" (UniqueName: \"kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-kube-api-access-phjvt\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:04.876694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.805444 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:04.876694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.805586 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:04.876694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.805649 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:04.876694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.805739 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phjvt\" (UniqueName: \"kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-kube-api-access-phjvt\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:04.876694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.806950 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:04.876694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.807694 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:04.897634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.877816 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:04.962659 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:04.931376 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phjvt\" (UniqueName: \"kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-kube-api-access-phjvt\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:05.110874 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:05.110841 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:05.668857 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:05.668833 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn"] Nov 03 14:33:05.674782 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:33:05.674754 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb381fe_de04_41da_a22d_c5a8ce304a89.slice/crio-fa56d2963e98886a6331c64f10cb514b5043b1ffaa3142e6b4a278ca49ae19bb WatchSource:0}: Error finding container fa56d2963e98886a6331c64f10cb514b5043b1ffaa3142e6b4a278ca49ae19bb: Status 404 returned error can't find the container with id fa56d2963e98886a6331c64f10cb514b5043b1ffaa3142e6b4a278ca49ae19bb Nov 03 14:33:06.672239 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:06.672209 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" event={"ID":"9bb381fe-de04-41da-a22d-c5a8ce304a89","Type":"ContainerStarted","Data":"a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb"} Nov 03 14:33:06.672681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:06.672258 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" event={"ID":"9bb381fe-de04-41da-a22d-c5a8ce304a89","Type":"ContainerStarted","Data":"fa56d2963e98886a6331c64f10cb514b5043b1ffaa3142e6b4a278ca49ae19bb"} Nov 03 14:33:06.672681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:06.672281 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:06.718518 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:06.718471 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" podStartSLOduration=2.718458291 podStartE2EDuration="2.718458291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:06.71583077 +0000 UTC m=+1262.274467278" watchObservedRunningTime="2026-11-03 14:33:06.718458291 +0000 UTC m=+1262.277094794" Nov 03 14:33:07.556993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:07.556957 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92"] Nov 03 14:33:12.582598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:12.582568 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" Nov 03 14:33:12.583052 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:12.582671 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" Nov 03 14:33:15.946465 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:15.946433 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs"] Nov 03 14:33:17.748428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:17.748398 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:17.777361 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:17.777326 6389 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b4361f/agc-cred-c88df00e610c: secret "agc-cred-c88df00e610c" not found Nov 03 14:33:17.777583 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:17.777568 6389 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn: secret "agc-cred-c88df00e610c" not found Nov 03 14:33:17.777816 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:17.777795 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials podName:9bb381fe-de04-41da-a22d-c5a8ce304a89 nodeName:}" failed. No retries permitted until 2026-11-03 14:33:18.277770177 +0000 UTC m=+1273.836406663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" (UID: "9bb381fe-de04-41da-a22d-c5a8ce304a89") : secret "agc-cred-c88df00e610c" not found Nov 03 14:33:18.246886 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.246860 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb"] Nov 03 14:33:18.252962 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.252942 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.270664 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.270635 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb"] Nov 03 14:33:18.316800 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:18.311878 6389 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b4361f/agc-cred-c88df00e610c: secret "agc-cred-c88df00e610c" not found Nov 03 14:33:18.316800 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:18.311908 6389 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn: secret "agc-cred-c88df00e610c" not found Nov 03 14:33:18.316800 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:18.312006 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials podName:9bb381fe-de04-41da-a22d-c5a8ce304a89 nodeName:}" failed. No retries permitted until 2026-11-03 14:33:19.311985452 +0000 UTC m=+1274.870621933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" (UID: "9bb381fe-de04-41da-a22d-c5a8ce304a89") : secret "agc-cred-c88df00e610c" not found Nov 03 14:33:18.419631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.416926 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw6s5\" (UniqueName: \"kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-kube-api-access-qw6s5\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.419631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.417009 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.419631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.417111 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.419631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.417177 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.518687 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.518493 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.518687 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.518627 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.518937 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.518841 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw6s5\" (UniqueName: \"kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-kube-api-access-qw6s5\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.518937 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.518886 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.522599 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.520097 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.522599 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.521115 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.524843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.524820 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.548186 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.548160 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw6s5\" (UniqueName: \"kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-kube-api-access-qw6s5\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.567565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.567539 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:18.877590 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:18.877560 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb"] Nov 03 14:33:18.879106 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:33:18.879079 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08fa1e6_5647_4296_89d3_ef16baa80ba5.slice/crio-c33eae91750d3c8d332ba8bb2320f63587e6e77a5c1e828c4f5c9af520c779fb WatchSource:0}: Error finding container c33eae91750d3c8d332ba8bb2320f63587e6e77a5c1e828c4f5c9af520c779fb: Status 404 returned error can't find the container with id c33eae91750d3c8d332ba8bb2320f63587e6e77a5c1e828c4f5c9af520c779fb Nov 03 14:33:19.129535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:19.129442 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" event={"ID":"a08fa1e6-5647-4296-89d3-ef16baa80ba5","Type":"ContainerStarted","Data":"bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9"} Nov 03 14:33:19.129535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:19.129510 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:19.129535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:19.129534 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" event={"ID":"a08fa1e6-5647-4296-89d3-ef16baa80ba5","Type":"ContainerStarted","Data":"c33eae91750d3c8d332ba8bb2320f63587e6e77a5c1e828c4f5c9af520c779fb"} Nov 03 14:33:19.180359 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:19.180297 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" podStartSLOduration=1.180278094 podStartE2EDuration="1.180278094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:19.17737762 +0000 UTC m=+1274.736014104" watchObservedRunningTime="2026-11-03 14:33:19.180278094 +0000 UTC m=+1274.738914579" Nov 03 14:33:19.329908 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:19.329870 6389 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b4361f/agc-cred-c88df00e610c: secret "agc-cred-c88df00e610c" not found Nov 03 14:33:19.329908 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:19.329898 6389 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn: secret "agc-cred-c88df00e610c" not found Nov 03 14:33:19.330132 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:19.329953 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials podName:9bb381fe-de04-41da-a22d-c5a8ce304a89 nodeName:}" failed. No retries permitted until 2026-11-03 14:33:21.329938449 +0000 UTC m=+1276.888574911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" (UID: "9bb381fe-de04-41da-a22d-c5a8ce304a89") : secret "agc-cred-c88df00e610c" not found Nov 03 14:33:20.947192 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:20.947049 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" podUID="31248427-a655-46ed-8578-97b7a207372c" containerName="payload-pre-processing" containerID="cri-o://a8dddc67070b6fa703d0c52a84f675bd6f8208468231ad061920e247acfcde7f" gracePeriod=25 Nov 03 14:33:21.149005 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:21.148965 6389 generic.go:356] "Generic (PLEG): container finished" podID="31248427-a655-46ed-8578-97b7a207372c" containerID="a8dddc67070b6fa703d0c52a84f675bd6f8208468231ad061920e247acfcde7f" exitCode=0 Nov 03 14:33:21.149178 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:21.149015 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" event={"ID":"31248427-a655-46ed-8578-97b7a207372c","Type":"ContainerDied","Data":"a8dddc67070b6fa703d0c52a84f675bd6f8208468231ad061920e247acfcde7f"} Nov 03 14:33:21.222085 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:21.222052 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" Nov 03 14:33:21.254163 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:21.254136 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31248427-a655-46ed-8578-97b7a207372c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31248427-a655-46ed-8578-97b7a207372c-plugins-config-volume\") pod \"31248427-a655-46ed-8578-97b7a207372c\" (UID: \"31248427-a655-46ed-8578-97b7a207372c\") " Nov 03 14:33:21.254342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:21.254322 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31248427-a655-46ed-8578-97b7a207372c-kube-api-access-dklz4\" (UniqueName: \"kubernetes.io/projected/31248427-a655-46ed-8578-97b7a207372c-kube-api-access-dklz4\") pod \"31248427-a655-46ed-8578-97b7a207372c\" (UID: \"31248427-a655-46ed-8578-97b7a207372c\") " Nov 03 14:33:21.254572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:21.254547 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31248427-a655-46ed-8578-97b7a207372c-plugins-config-volume" pod "31248427-a655-46ed-8578-97b7a207372c" (UID: "31248427-a655-46ed-8578-97b7a207372c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:21.255054 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:21.254968 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31248427-a655-46ed-8578-97b7a207372c-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:21.256672 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:21.256594 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31248427-a655-46ed-8578-97b7a207372c-kube-api-access-dklz4" pod "31248427-a655-46ed-8578-97b7a207372c" (UID: "31248427-a655-46ed-8578-97b7a207372c"). InnerVolumeSpecName "kube-api-access-dklz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:21.365847 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:21.363038 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dklz4\" (UniqueName: \"kubernetes.io/projected/31248427-a655-46ed-8578-97b7a207372c-kube-api-access-dklz4\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:21.365847 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:21.363159 6389 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b4361f/agc-cred-c88df00e610c: secret "agc-cred-c88df00e610c" not found Nov 03 14:33:21.365847 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:21.363184 6389 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn: secret "agc-cred-c88df00e610c" not found Nov 03 14:33:21.365847 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:21.363255 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials podName:9bb381fe-de04-41da-a22d-c5a8ce304a89 nodeName:}" failed. No retries permitted until 2026-11-03 14:33:25.363236982 +0000 UTC m=+1280.921873467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" (UID: "9bb381fe-de04-41da-a22d-c5a8ce304a89") : secret "agc-cred-c88df00e610c" not found Nov 03 14:33:22.157075 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.157043 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" Nov 03 14:33:22.157527 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.157075 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs" event={"ID":"31248427-a655-46ed-8578-97b7a207372c","Type":"ContainerDied","Data":"52ed90998db6787d19bf91866a825661770170d9519020044a5a7dc30235737b"} Nov 03 14:33:22.157527 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.157146 6389 scope.go:122] "RemoveContainer" containerID="a8dddc67070b6fa703d0c52a84f675bd6f8208468231ad061920e247acfcde7f" Nov 03 14:33:22.213767 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.213743 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs"] Nov 03 14:33:22.227462 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.227440 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-vgvbs"] Nov 03 14:33:22.592941 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.592903 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" podUID="d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6" containerName="main" containerID="cri-o://a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a" gracePeriod=45 Nov 03 14:33:22.620437 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.620412 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31248427-a655-46ed-8578-97b7a207372c" path="/var/lib/kubelet/pods/31248427-a655-46ed-8578-97b7a207372c/volumes" Nov 03 14:33:22.869886 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.869864 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:33:22.885030 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.885005 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-dshm\") pod \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " Nov 03 14:33:22.885165 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.885102 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-kube-api-access-gdntv\" (UniqueName: \"kubernetes.io/projected/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-kube-api-access-gdntv\") pod \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " Nov 03 14:33:22.885165 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.885131 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-home\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-home\") pod \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " Nov 03 14:33:22.885275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.885179 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-model-cache\") pod \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " Nov 03 14:33:22.885275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.885226 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tls-certs\") pod \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " Nov 03 14:33:22.885387 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.885350 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tmp-dir\") pod \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\" (UID: \"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6\") " Nov 03 14:33:22.885484 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.885432 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-home" pod "d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6" (UID: "d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:22.885656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.885631 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-model-cache" pod "d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6" (UID: "d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:22.885907 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.885871 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tmp-dir" pod "d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6" (UID: "d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:22.886077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.886058 6389 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tmp-dir\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:22.886153 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.886082 6389 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-home\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:22.886153 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.886096 6389 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-model-cache\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:22.887354 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.887327 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-dshm" pod "d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6" (UID: "d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:22.887714 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.887692 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tls-certs" pod "d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6" (UID: "d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:33:22.887888 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.887863 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-kube-api-access-gdntv" pod "d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6" (UID: "d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6"). InnerVolumeSpecName "kube-api-access-gdntv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:22.988418 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.988373 6389 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-tls-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:22.988418 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.988405 6389 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-dshm\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:22.988418 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:22.988419 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdntv\" (UniqueName: \"kubernetes.io/projected/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6-kube-api-access-gdntv\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:23.172284 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:23.172255 6389 generic.go:356] "Generic (PLEG): container finished" podID="d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6" containerID="a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a" exitCode=0 Nov 03 14:33:23.172760 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:23.172301 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" Nov 03 14:33:23.172760 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:23.172309 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" event={"ID":"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6","Type":"ContainerDied","Data":"a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a"} Nov 03 14:33:23.172760 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:23.172339 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92" event={"ID":"d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6","Type":"ContainerDied","Data":"06dfd027d6e6ae0d1759bb17281b68934c4ed0ae5212dd94f648d86add3d0b81"} Nov 03 14:33:23.172760 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:23.172355 6389 scope.go:122] "RemoveContainer" containerID="a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a" Nov 03 14:33:23.182669 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:23.182652 6389 scope.go:122] "RemoveContainer" containerID="a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a" Nov 03 14:33:23.182937 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:23.182919 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a\": container with ID starting with a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a not found: ID does not exist" containerID="a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a" Nov 03 14:33:23.182987 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:23.182946 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a"} err="failed to get container status \"a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a\": rpc error: code = NotFound desc = could not find container \"a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a\": container with ID starting with a94970bf4bed2d76cbb1cd90ca59f99ed060ec5471b13666a9c610b92cef0a6a not found: ID does not exist" Nov 03 14:33:23.231067 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:23.231041 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92"] Nov 03 14:33:23.250765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:23.250738 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-6653eb/autores-model-6653eb-kserve-587557877c-zqh92"] Nov 03 14:33:24.618771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:24.618738 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6" path="/var/lib/kubelet/pods/d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6/volumes" Nov 03 14:33:25.438632 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:25.436234 6389 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b4361f/agc-cred-c88df00e610c: secret "agc-cred-c88df00e610c" not found Nov 03 14:33:25.438632 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:25.436267 6389 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn: secret "agc-cred-c88df00e610c" not found Nov 03 14:33:25.438632 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:25.436347 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials podName:9bb381fe-de04-41da-a22d-c5a8ce304a89 nodeName:}" failed. No retries permitted until 2026-11-03 14:33:33.436329327 +0000 UTC m=+1288.994965788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" (UID: "9bb381fe-de04-41da-a22d-c5a8ce304a89") : secret "agc-cred-c88df00e610c" not found Nov 03 14:33:28.375334 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:28.375296 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:33:28.688316 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:28.688277 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn"] Nov 03 14:33:28.702135 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:28.702104 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb"] Nov 03 14:33:28.705698 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:28.705671 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:31.294708 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.294674 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw"] Nov 03 14:33:31.295454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.295431 6389 cpu_manager.go:414] "containerMap: removing container" podUID="31248427-a655-46ed-8578-97b7a207372c" containerName="payload-pre-processing" Nov 03 14:33:31.295454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.295452 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31248427-a655-46ed-8578-97b7a207372c" containerName="payload-pre-processing" Nov 03 14:33:31.295454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.295469 6389 cpu_manager.go:414] "containerMap: removing container" podUID="d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6" containerName="main" Nov 03 14:33:31.295711 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.295476 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6" containerName="main" Nov 03 14:33:31.295711 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.295582 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b26ed3-e0d9-4fe1-83d4-ad1c47ee65b6" containerName="main" Nov 03 14:33:31.295711 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.295599 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="31248427-a655-46ed-8578-97b7a207372c" containerName="payload-pre-processing" Nov 03 14:33:31.300835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.300815 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.336972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.336946 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw"] Nov 03 14:33:31.437566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.437525 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.437767 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.437593 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.437767 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.437711 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.437891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.437858 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmqw8\" (UniqueName: \"kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-kube-api-access-dmqw8\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.437962 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.437938 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.438079 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.438059 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.438238 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.438222 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.438276 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.438264 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.438336 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.438319 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.539990 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.539957 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.540189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.540009 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.540189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.540137 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.540189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.540187 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.540360 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.540239 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.540360 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.540255 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.540360 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.540292 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqw8\" (UniqueName: \"kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-kube-api-access-dmqw8\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.540360 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.540351 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.540573 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.540426 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.540573 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.540554 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.540739 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.540711 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.540854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.540836 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.540995 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.540977 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.541072 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.541041 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.542373 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.542358 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.543062 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.543044 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.562653 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.562560 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmqw8\" (UniqueName: \"kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-kube-api-access-dmqw8\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.562653 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.562579 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.615013 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.614987 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:31.945257 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.945233 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw"] Nov 03 14:33:31.946718 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:33:31.946693 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fedcbdd_19f2_4b86_a328_3431ee04dc75.slice/crio-468dea942e03dade6a1c744bf6b9fd7eb83ebd7713034b1783733f5fb1ce294b WatchSource:0}: Error finding container 468dea942e03dade6a1c744bf6b9fd7eb83ebd7713034b1783733f5fb1ce294b: Status 404 returned error can't find the container with id 468dea942e03dade6a1c744bf6b9fd7eb83ebd7713034b1783733f5fb1ce294b Nov 03 14:33:31.949077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.949050 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:33:31.949145 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.949115 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:33:31.949182 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:31.949141 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:33:32.271565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:32.266857 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" event={"ID":"5fedcbdd-19f2-4b86-a328-3431ee04dc75","Type":"ContainerStarted","Data":"1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020"} Nov 03 14:33:32.271565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:32.266930 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" event={"ID":"5fedcbdd-19f2-4b86-a328-3431ee04dc75","Type":"ContainerStarted","Data":"468dea942e03dade6a1c744bf6b9fd7eb83ebd7713034b1783733f5fb1ce294b"} Nov 03 14:33:32.329680 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:32.329263 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" podStartSLOduration=1.32924372 podStartE2EDuration="1.32924372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:32.328879041 +0000 UTC m=+1287.887515526" watchObservedRunningTime="2026-11-03 14:33:32.32924372 +0000 UTC m=+1287.887880205" Nov 03 14:33:32.632691 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:32.625985 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:33.484413 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:33.484381 6389 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b4361f/agc-cred-95ac97e0ca1e: secret "agc-cred-95ac97e0ca1e" not found Nov 03 14:33:33.484413 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:33.484406 6389 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b4361f/agc-cred-c88df00e610c: secret "agc-cred-c88df00e610c" not found Nov 03 14:33:33.484413 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:33.484417 6389 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn: [secret "agc-cred-95ac97e0ca1e" not found, secret "agc-cred-c88df00e610c" not found] Nov 03 14:33:33.484933 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:33.484471 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials podName:9bb381fe-de04-41da-a22d-c5a8ce304a89 nodeName:}" failed. No retries permitted until 2026-11-03 14:33:49.484457833 +0000 UTC m=+1305.043094309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" (UID: "9bb381fe-de04-41da-a22d-c5a8ce304a89") : [secret "agc-cred-95ac97e0ca1e" not found, secret "agc-cred-c88df00e610c" not found] Nov 03 14:33:33.617526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:33.617494 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:33:33.617738 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:33.617543 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" podUID="5fedcbdd-19f2-4b86-a328-3431ee04dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 14:33:33.689347 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:33.689312 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" podUID="9bb381fe-de04-41da-a22d-c5a8ce304a89" containerName="payload-processing" containerID="cri-o://a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb" gracePeriod=25 Nov 03 14:33:33.702889 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:33.702843 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" podUID="a08fa1e6-5647-4296-89d3-ef16baa80ba5" containerName="payload-processing" containerID="cri-o://bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9" gracePeriod=25 Nov 03 14:33:34.154917 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.154879 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:34.157196 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.157171 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:34.249698 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.195977 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-kube-api-access-phjvt\" (UniqueName: \"kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-kube-api-access-phjvt\") pod \"9bb381fe-de04-41da-a22d-c5a8ce304a89\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " Nov 03 14:33:34.250006 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.249981 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-plugins-config-volume\") pod \"9bb381fe-de04-41da-a22d-c5a8ce304a89\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " Nov 03 14:33:34.250134 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.250120 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials\") pod \"9bb381fe-de04-41da-a22d-c5a8ce304a89\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " Nov 03 14:33:34.258825 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.250405 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-routing-overlay-volume\") pod \"9bb381fe-de04-41da-a22d-c5a8ce304a89\" (UID: \"9bb381fe-de04-41da-a22d-c5a8ce304a89\") " Nov 03 14:33:34.273356 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.252074 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-routing-overlay-volume" pod "9bb381fe-de04-41da-a22d-c5a8ce304a89" (UID: "9bb381fe-de04-41da-a22d-c5a8ce304a89"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:34.278856 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.255756 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-kube-api-access-phjvt" pod "9bb381fe-de04-41da-a22d-c5a8ce304a89" (UID: "9bb381fe-de04-41da-a22d-c5a8ce304a89"). InnerVolumeSpecName "kube-api-access-phjvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:34.279069 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.255890 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-plugins-config-volume" pod "9bb381fe-de04-41da-a22d-c5a8ce304a89" (UID: "9bb381fe-de04-41da-a22d-c5a8ce304a89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:34.313859 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.313792 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials" pod "9bb381fe-de04-41da-a22d-c5a8ce304a89" (UID: "9bb381fe-de04-41da-a22d-c5a8ce304a89"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:34.372586 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.372489 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-plugins-config-volume\") pod \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " Nov 03 14:33:34.372884 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.372864 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-kube-api-access-qw6s5\" (UniqueName: \"kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-kube-api-access-qw6s5\") pod \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " Nov 03 14:33:34.373123 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.373103 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-provider-credentials\") pod \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " Nov 03 14:33:34.373442 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.373425 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-routing-overlay-volume\") pod \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\" (UID: \"a08fa1e6-5647-4296-89d3-ef16baa80ba5\") " Nov 03 14:33:34.374122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.374099 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phjvt\" (UniqueName: \"kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-kube-api-access-phjvt\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:34.374289 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.374273 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:34.374392 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.374380 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bb381fe-de04-41da-a22d-c5a8ce304a89-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:34.374480 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.374467 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bb381fe-de04-41da-a22d-c5a8ce304a89-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:34.374946 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.374922 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-routing-overlay-volume" pod "a08fa1e6-5647-4296-89d3-ef16baa80ba5" (UID: "a08fa1e6-5647-4296-89d3-ef16baa80ba5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:34.375486 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.375464 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-plugins-config-volume" pod "a08fa1e6-5647-4296-89d3-ef16baa80ba5" (UID: "a08fa1e6-5647-4296-89d3-ef16baa80ba5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:34.381678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.379759 6389 generic.go:356] "Generic (PLEG): container finished" podID="a08fa1e6-5647-4296-89d3-ef16baa80ba5" containerID="bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9" exitCode=0 Nov 03 14:33:34.381678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.379872 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" event={"ID":"a08fa1e6-5647-4296-89d3-ef16baa80ba5","Type":"ContainerDied","Data":"bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9"} Nov 03 14:33:34.381678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.380008 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" event={"ID":"a08fa1e6-5647-4296-89d3-ef16baa80ba5","Type":"ContainerDied","Data":"c33eae91750d3c8d332ba8bb2320f63587e6e77a5c1e828c4f5c9af520c779fb"} Nov 03 14:33:34.381678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.380043 6389 scope.go:122] "RemoveContainer" containerID="bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9" Nov 03 14:33:34.381678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.380208 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb" Nov 03 14:33:34.385703 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.385585 6389 generic.go:356] "Generic (PLEG): container finished" podID="9bb381fe-de04-41da-a22d-c5a8ce304a89" containerID="a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb" exitCode=0 Nov 03 14:33:34.387113 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.387095 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" Nov 03 14:33:34.390357 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.390318 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" event={"ID":"9bb381fe-de04-41da-a22d-c5a8ce304a89","Type":"ContainerDied","Data":"a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb"} Nov 03 14:33:34.390455 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.390388 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn" event={"ID":"9bb381fe-de04-41da-a22d-c5a8ce304a89","Type":"ContainerDied","Data":"fa56d2963e98886a6331c64f10cb514b5043b1ffaa3142e6b4a278ca49ae19bb"} Nov 03 14:33:34.414542 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.414492 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-kube-api-access-qw6s5" pod "a08fa1e6-5647-4296-89d3-ef16baa80ba5" (UID: "a08fa1e6-5647-4296-89d3-ef16baa80ba5"). InnerVolumeSpecName "kube-api-access-qw6s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:34.425159 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.424824 6389 scope.go:122] "RemoveContainer" containerID="bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9" Nov 03 14:33:34.425582 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:34.425404 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9\": container with ID starting with bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9 not found: ID does not exist" containerID="bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9" Nov 03 14:33:34.425582 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.425444 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9"} err="failed to get container status \"bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9\": rpc error: code = NotFound desc = could not find container \"bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9\": container with ID starting with bd776cabf6f65192feeb3486eeff0e8046a42c0b2dcb79640ffcf0342a6c3aa9 not found: ID does not exist" Nov 03 14:33:34.425582 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.425469 6389 scope.go:122] "RemoveContainer" containerID="a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb" Nov 03 14:33:34.427645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.427590 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-provider-credentials" pod "a08fa1e6-5647-4296-89d3-ef16baa80ba5" (UID: "a08fa1e6-5647-4296-89d3-ef16baa80ba5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:34.461557 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.461514 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn"] Nov 03 14:33:34.467800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.467774 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-75b995mgn"] Nov 03 14:33:34.471094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.471075 6389 scope.go:122] "RemoveContainer" containerID="a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb" Nov 03 14:33:34.471501 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:34.471474 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb\": container with ID starting with a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb not found: ID does not exist" containerID="a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb" Nov 03 14:33:34.471592 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.471511 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb"} err="failed to get container status \"a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb\": rpc error: code = NotFound desc = could not find container \"a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb\": container with ID starting with a9d8cbb47ba05e8d504e9aaac02deb40b646f808901fb971a7b4c98623f69adb not found: ID does not exist" Nov 03 14:33:34.476449 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.476423 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:34.476449 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.476449 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08fa1e6-5647-4296-89d3-ef16baa80ba5-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:34.476593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.476458 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw6s5\" (UniqueName: \"kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-kube-api-access-qw6s5\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:34.476593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.476468 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a08fa1e6-5647-4296-89d3-ef16baa80ba5-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:34.616217 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.616175 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:33:34.616778 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.616238 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" podUID="5fedcbdd-19f2-4b86-a328-3431ee04dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 14:33:34.643699 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.641403 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb381fe-de04-41da-a22d-c5a8ce304a89" path="/var/lib/kubelet/pods/9bb381fe-de04-41da-a22d-c5a8ce304a89/volumes" Nov 03 14:33:34.738968 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.738926 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb"] Nov 03 14:33:34.739248 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:34.739230 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5smqfb"] Nov 03 14:33:35.615644 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:35.615596 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:33:35.615815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:35.615692 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" podUID="5fedcbdd-19f2-4b86-a328-3431ee04dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 14:33:36.616162 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:36.616131 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:33:36.616555 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:36.616190 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" podUID="5fedcbdd-19f2-4b86-a328-3431ee04dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 14:33:36.619656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:36.619450 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08fa1e6-5647-4296-89d3-ef16baa80ba5" path="/var/lib/kubelet/pods/a08fa1e6-5647-4296-89d3-ef16baa80ba5/volumes" Nov 03 14:33:37.615664 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:37.615628 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:33:37.615869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:37.615692 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" podUID="5fedcbdd-19f2-4b86-a328-3431ee04dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 14:33:39.048302 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:39.048267 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:39.049279 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:39.049252 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:39.050339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:39.050315 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:33:47.530872 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.530834 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5"] Nov 03 14:33:47.531774 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.531754 6389 cpu_manager.go:414] "containerMap: removing container" podUID="a08fa1e6-5647-4296-89d3-ef16baa80ba5" containerName="payload-processing" Nov 03 14:33:47.531774 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.531775 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08fa1e6-5647-4296-89d3-ef16baa80ba5" containerName="payload-processing" Nov 03 14:33:47.531952 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.531825 6389 cpu_manager.go:414] "containerMap: removing container" podUID="9bb381fe-de04-41da-a22d-c5a8ce304a89" containerName="payload-processing" Nov 03 14:33:47.531952 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.531832 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb381fe-de04-41da-a22d-c5a8ce304a89" containerName="payload-processing" Nov 03 14:33:47.532052 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.531990 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb381fe-de04-41da-a22d-c5a8ce304a89" containerName="payload-processing" Nov 03 14:33:47.532052 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.532008 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08fa1e6-5647-4296-89d3-ef16baa80ba5" containerName="payload-processing" Nov 03 14:33:47.537559 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.537538 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.542946 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.542924 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hz8k\" (UniqueName: \"kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-kube-api-access-2hz8k\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.543085 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.543066 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.543217 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.543204 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.543260 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.543233 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.568210 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.568180 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"e2e-cmp-legacy-d5ee9e-provider-cred\"" Nov 03 14:33:47.568428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.568202 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-331c9e\"" Nov 03 14:33:47.568428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.568214 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"kube-root-ca.crt\"" Nov 03 14:33:47.568428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.568253 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"openshift-service-ca.crt\"" Nov 03 14:33:47.568428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.568261 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-e2e--e9e6f325-dockercfg-nxpfd\"" Nov 03 14:33:47.568712 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.568295 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"routing-overlay\"" Nov 03 14:33:47.571387 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.571361 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5"] Nov 03 14:33:47.644338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.644300 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hz8k\" (UniqueName: \"kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-kube-api-access-2hz8k\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.644511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.644409 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.644579 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.644563 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.644736 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.644628 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.645194 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.645170 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.645282 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.645272 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.647268 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.647248 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.667007 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.666973 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hz8k\" (UniqueName: \"kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-kube-api-access-2hz8k\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:47.851570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:47.851254 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:48.241031 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:33:48.241000 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31f97c3_0499_4900_bd83_9c0cfe4a3c5c.slice/crio-d29f97264cb6239f52d20c1aeee9c25c167df9afeffef3306f6c133a9bb60259 WatchSource:0}: Error finding container d29f97264cb6239f52d20c1aeee9c25c167df9afeffef3306f6c133a9bb60259: Status 404 returned error can't find the container with id d29f97264cb6239f52d20c1aeee9c25c167df9afeffef3306f6c133a9bb60259 Nov 03 14:33:48.241031 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:48.241006 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5"] Nov 03 14:33:48.525306 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:48.525210 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" event={"ID":"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c","Type":"ContainerStarted","Data":"9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81"} Nov 03 14:33:48.525306 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:48.525269 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" event={"ID":"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c","Type":"ContainerStarted","Data":"d29f97264cb6239f52d20c1aeee9c25c167df9afeffef3306f6c133a9bb60259"} Nov 03 14:33:48.525306 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:48.525296 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:48.571116 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:48.571065 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" podStartSLOduration=1.571052039 podStartE2EDuration="1.571052039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:48.569767571 +0000 UTC m=+1304.128404054" watchObservedRunningTime="2026-11-03 14:33:48.571052039 +0000 UTC m=+1304.129688522" Nov 03 14:33:50.090802 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:50.090764 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85"] Nov 03 14:33:54.097245 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:54.097214 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql"] Nov 03 14:33:54.123017 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:54.122985 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6"] Nov 03 14:33:55.092007 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.091962 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" podUID="e6ae7db7-52d6-41fd-b4c3-0cc19e834f60" containerName="payload-processing" containerID="cri-o://1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1" gracePeriod=25 Nov 03 14:33:55.514549 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.514524 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" Nov 03 14:33:55.543910 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.543882 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-plugins-config-volume\") pod \"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60\" (UID: \"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60\") " Nov 03 14:33:55.544101 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.543974 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-kube-api-access-l8tjv\" (UniqueName: \"kubernetes.io/projected/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-kube-api-access-l8tjv\") pod \"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60\" (UID: \"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60\") " Nov 03 14:33:55.544879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.544855 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-plugins-config-volume" pod "e6ae7db7-52d6-41fd-b4c3-0cc19e834f60" (UID: "e6ae7db7-52d6-41fd-b4c3-0cc19e834f60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:55.546783 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.546756 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-kube-api-access-l8tjv" pod "e6ae7db7-52d6-41fd-b4c3-0cc19e834f60" (UID: "e6ae7db7-52d6-41fd-b4c3-0cc19e834f60"). InnerVolumeSpecName "kube-api-access-l8tjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:55.608699 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.608657 6389 generic.go:356] "Generic (PLEG): container finished" podID="e6ae7db7-52d6-41fd-b4c3-0cc19e834f60" containerID="1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1" exitCode=0 Nov 03 14:33:55.608862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.608711 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" event={"ID":"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60","Type":"ContainerDied","Data":"1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1"} Nov 03 14:33:55.608862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.608744 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" Nov 03 14:33:55.608862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.608798 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85" event={"ID":"e6ae7db7-52d6-41fd-b4c3-0cc19e834f60","Type":"ContainerDied","Data":"c42b44e00d91eb385cf0bdd1b12db2dd72a143cea1c8eec4a544d9e9ee0631e3"} Nov 03 14:33:55.608862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.608830 6389 scope.go:122] "RemoveContainer" containerID="1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1" Nov 03 14:33:55.644542 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.644420 6389 scope.go:122] "RemoveContainer" containerID="1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1" Nov 03 14:33:55.645986 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:55.645037 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1\": container with ID starting with 1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1 not found: ID does not exist" containerID="1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1" Nov 03 14:33:55.645986 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.645085 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1"} err="failed to get container status \"1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1\": rpc error: code = NotFound desc = could not find container \"1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1\": container with ID starting with 1aa70cc7af12c1f29164ed8db082ac4c9876ecfc79aac164a77488d993a2f2a1 not found: ID does not exist" Nov 03 14:33:55.646876 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.646812 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8tjv\" (UniqueName: \"kubernetes.io/projected/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-kube-api-access-l8tjv\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:55.646876 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.646840 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:55.689580 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.689546 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85"] Nov 03 14:33:55.697388 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:55.697357 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6653eb-6d77d87d6c-mbh85"] Nov 03 14:33:56.619232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:56.619199 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ae7db7-52d6-41fd-b4c3-0cc19e834f60" path="/var/lib/kubelet/pods/e6ae7db7-52d6-41fd-b4c3-0cc19e834f60/volumes" Nov 03 14:33:58.383153 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:58.383122 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:33:59.098096 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.097971 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" podUID="07b326a2-b5e6-4030-b9bd-4d85d19107c3" containerName="payload-processing" containerID="cri-o://51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274" gracePeriod=25 Nov 03 14:33:59.123263 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.123235 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" podUID="3676abfd-9064-436e-83be-4aa79bf35f01" containerName="payload-pre-processing" containerID="cri-o://d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70" gracePeriod=25 Nov 03 14:33:59.152031 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:59.152004 6389 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b326a2_b5e6_4030_b9bd_4d85d19107c3.slice/crio-conmon-51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274.scope\": RecentStats: unable to find data in memory cache]" Nov 03 14:33:59.152157 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:59.152055 6389 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b326a2_b5e6_4030_b9bd_4d85d19107c3.slice/crio-conmon-51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274.scope\": RecentStats: unable to find data in memory cache]" Nov 03 14:33:59.407345 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.407316 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" Nov 03 14:33:59.466387 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.466360 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" Nov 03 14:33:59.575633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.571238 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b326a2-b5e6-4030-b9bd-4d85d19107c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b326a2-b5e6-4030-b9bd-4d85d19107c3-plugins-config-volume\") pod \"07b326a2-b5e6-4030-b9bd-4d85d19107c3\" (UID: \"07b326a2-b5e6-4030-b9bd-4d85d19107c3\") " Nov 03 14:33:59.575633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.571345 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3676abfd-9064-436e-83be-4aa79bf35f01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3676abfd-9064-436e-83be-4aa79bf35f01-plugins-config-volume\") pod \"3676abfd-9064-436e-83be-4aa79bf35f01\" (UID: \"3676abfd-9064-436e-83be-4aa79bf35f01\") " Nov 03 14:33:59.575633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.571413 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b326a2-b5e6-4030-b9bd-4d85d19107c3-kube-api-access-hjfxw\" (UniqueName: \"kubernetes.io/projected/07b326a2-b5e6-4030-b9bd-4d85d19107c3-kube-api-access-hjfxw\") pod \"07b326a2-b5e6-4030-b9bd-4d85d19107c3\" (UID: \"07b326a2-b5e6-4030-b9bd-4d85d19107c3\") " Nov 03 14:33:59.575633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.571540 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3676abfd-9064-436e-83be-4aa79bf35f01-kube-api-access-nvgcz\" (UniqueName: \"kubernetes.io/projected/3676abfd-9064-436e-83be-4aa79bf35f01-kube-api-access-nvgcz\") pod \"3676abfd-9064-436e-83be-4aa79bf35f01\" (UID: \"3676abfd-9064-436e-83be-4aa79bf35f01\") " Nov 03 14:33:59.651730 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.599109 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3676abfd-9064-436e-83be-4aa79bf35f01-plugins-config-volume" pod "3676abfd-9064-436e-83be-4aa79bf35f01" (UID: "3676abfd-9064-436e-83be-4aa79bf35f01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:59.651730 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.599554 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b326a2-b5e6-4030-b9bd-4d85d19107c3-plugins-config-volume" pod "07b326a2-b5e6-4030-b9bd-4d85d19107c3" (UID: "07b326a2-b5e6-4030-b9bd-4d85d19107c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:59.718018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.678761 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b326a2-b5e6-4030-b9bd-4d85d19107c3-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:59.718018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.678795 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3676abfd-9064-436e-83be-4aa79bf35f01-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:59.718018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.716795 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:33:59.730147 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.728330 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b326a2-b5e6-4030-b9bd-4d85d19107c3-kube-api-access-hjfxw" pod "07b326a2-b5e6-4030-b9bd-4d85d19107c3" (UID: "07b326a2-b5e6-4030-b9bd-4d85d19107c3"). InnerVolumeSpecName "kube-api-access-hjfxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:59.730302 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.730240 6389 generic.go:356] "Generic (PLEG): container finished" podID="3676abfd-9064-436e-83be-4aa79bf35f01" containerID="d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70" exitCode=0 Nov 03 14:33:59.730967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.730476 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" Nov 03 14:33:59.730967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.730717 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" event={"ID":"3676abfd-9064-436e-83be-4aa79bf35f01","Type":"ContainerDied","Data":"d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70"} Nov 03 14:33:59.730967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.730749 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3676abfd-9064-436e-83be-4aa79bf35f01-kube-api-access-nvgcz" pod "3676abfd-9064-436e-83be-4aa79bf35f01" (UID: "3676abfd-9064-436e-83be-4aa79bf35f01"). InnerVolumeSpecName "kube-api-access-nvgcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:59.730967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.730769 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6" event={"ID":"3676abfd-9064-436e-83be-4aa79bf35f01","Type":"ContainerDied","Data":"9cf7f0fe5f177694aa33e36b31fe09fd30262ae27997ea9f669784bc06c5ba58"} Nov 03 14:33:59.730967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.730801 6389 scope.go:122] "RemoveContainer" containerID="d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70" Nov 03 14:33:59.736645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.735298 6389 generic.go:356] "Generic (PLEG): container finished" podID="07b326a2-b5e6-4030-b9bd-4d85d19107c3" containerID="51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274" exitCode=0 Nov 03 14:33:59.736645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.735354 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" event={"ID":"07b326a2-b5e6-4030-b9bd-4d85d19107c3","Type":"ContainerDied","Data":"51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274"} Nov 03 14:33:59.736645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.735400 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" event={"ID":"07b326a2-b5e6-4030-b9bd-4d85d19107c3","Type":"ContainerDied","Data":"b79aa3ab6477141cbadd226745e4e1e0df54936a9763318ead6ee43c177d6461"} Nov 03 14:33:59.736645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.735480 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql" Nov 03 14:33:59.752254 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.752230 6389 scope.go:122] "RemoveContainer" containerID="d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70" Nov 03 14:33:59.752574 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:59.752552 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70\": container with ID starting with d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70 not found: ID does not exist" containerID="d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70" Nov 03 14:33:59.752692 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.752582 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70"} err="failed to get container status \"d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70\": rpc error: code = NotFound desc = could not find container \"d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70\": container with ID starting with d2e460600fa833ecb2c37179c1fcc305c530d09ee98924680de2c2e9c87d9f70 not found: ID does not exist" Nov 03 14:33:59.752692 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.752601 6389 scope.go:122] "RemoveContainer" containerID="51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274" Nov 03 14:33:59.762272 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.762250 6389 scope.go:122] "RemoveContainer" containerID="51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274" Nov 03 14:33:59.762544 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:33:59.762526 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274\": container with ID starting with 51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274 not found: ID does not exist" containerID="51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274" Nov 03 14:33:59.762684 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.762554 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274"} err="failed to get container status \"51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274\": rpc error: code = NotFound desc = could not find container \"51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274\": container with ID starting with 51c7c102349f1a39640beab0f54865460e799429f3fc9c63118019bfb02a4274 not found: ID does not exist" Nov 03 14:33:59.780678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.780592 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjfxw\" (UniqueName: \"kubernetes.io/projected/07b326a2-b5e6-4030-b9bd-4d85d19107c3-kube-api-access-hjfxw\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:59.780678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.780663 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvgcz\" (UniqueName: \"kubernetes.io/projected/3676abfd-9064-436e-83be-4aa79bf35f01-kube-api-access-nvgcz\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:33:59.817484 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.817455 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql"] Nov 03 14:33:59.824836 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:33:59.824815 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-95af53a1-86bd5fb577-gfnql"] Nov 03 14:34:00.079836 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:00.079805 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6"] Nov 03 14:34:00.086359 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:00.086332 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-95af53a1-6ff6cd6d55-2xcv6"] Nov 03 14:34:00.620273 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:00.620239 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b326a2-b5e6-4030-b9bd-4d85d19107c3" path="/var/lib/kubelet/pods/07b326a2-b5e6-4030-b9bd-4d85d19107c3/volumes" Nov 03 14:34:00.620874 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:00.620857 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3676abfd-9064-436e-83be-4aa79bf35f01" path="/var/lib/kubelet/pods/3676abfd-9064-436e-83be-4aa79bf35f01/volumes" Nov 03 14:34:05.269120 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.269075 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7"] Nov 03 14:34:05.269673 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.269659 6389 cpu_manager.go:414] "containerMap: removing container" podUID="e6ae7db7-52d6-41fd-b4c3-0cc19e834f60" containerName="payload-processing" Nov 03 14:34:05.269673 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.269673 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ae7db7-52d6-41fd-b4c3-0cc19e834f60" containerName="payload-processing" Nov 03 14:34:05.269763 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.269687 6389 cpu_manager.go:414] "containerMap: removing container" podUID="07b326a2-b5e6-4030-b9bd-4d85d19107c3" containerName="payload-processing" Nov 03 14:34:05.269763 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.269693 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b326a2-b5e6-4030-b9bd-4d85d19107c3" containerName="payload-processing" Nov 03 14:34:05.269763 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.269709 6389 cpu_manager.go:414] "containerMap: removing container" podUID="3676abfd-9064-436e-83be-4aa79bf35f01" containerName="payload-pre-processing" Nov 03 14:34:05.269763 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.269714 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3676abfd-9064-436e-83be-4aa79bf35f01" containerName="payload-pre-processing" Nov 03 14:34:05.269887 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.269791 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b326a2-b5e6-4030-b9bd-4d85d19107c3" containerName="payload-processing" Nov 03 14:34:05.269887 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.269802 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="3676abfd-9064-436e-83be-4aa79bf35f01" containerName="payload-pre-processing" Nov 03 14:34:05.269887 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.269812 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ae7db7-52d6-41fd-b4c3-0cc19e834f60" containerName="payload-processing" Nov 03 14:34:05.274696 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.274666 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" Nov 03 14:34:05.295622 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.295575 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7"] Nov 03 14:34:05.304862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.304839 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cb587c-dockercfg-5vfls\"" Nov 03 14:34:05.304862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.304837 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cb587c\"" Nov 03 14:34:05.369966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.369933 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8cml\" (UniqueName: \"kubernetes.io/projected/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-kube-api-access-r8cml\") pod \"payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7\" (UID: \"55e3b5f4-8a45-4c0e-a370-f37e9da34eda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" Nov 03 14:34:05.370150 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.370101 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7\" (UID: \"55e3b5f4-8a45-4c0e-a370-f37e9da34eda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" Nov 03 14:34:05.454411 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.450683 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq"] Nov 03 14:34:05.479062 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.479016 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7\" (UID: \"55e3b5f4-8a45-4c0e-a370-f37e9da34eda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" Nov 03 14:34:05.479262 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.479246 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8cml\" (UniqueName: \"kubernetes.io/projected/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-kube-api-access-r8cml\") pod \"payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7\" (UID: \"55e3b5f4-8a45-4c0e-a370-f37e9da34eda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" Nov 03 14:34:05.485645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.480627 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7\" (UID: \"55e3b5f4-8a45-4c0e-a370-f37e9da34eda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" Nov 03 14:34:05.614275 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.614173 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq"] Nov 03 14:34:05.615262 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.615238 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" Nov 03 14:34:05.617001 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.616950 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8cml\" (UniqueName: \"kubernetes.io/projected/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-kube-api-access-r8cml\") pod \"payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7\" (UID: \"55e3b5f4-8a45-4c0e-a370-f37e9da34eda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" Nov 03 14:34:05.681947 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.681903 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pth6\" (UniqueName: \"kubernetes.io/projected/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-kube-api-access-2pth6\") pod \"payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq\" (UID: \"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1\") " pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" Nov 03 14:34:05.682139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.682087 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq\" (UID: \"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1\") " pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" Nov 03 14:34:05.783856 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.783824 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pth6\" (UniqueName: \"kubernetes.io/projected/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-kube-api-access-2pth6\") pod \"payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq\" (UID: \"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1\") " pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" Nov 03 14:34:05.784065 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.783880 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq\" (UID: \"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1\") " pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" Nov 03 14:34:05.784522 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.784502 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq\" (UID: \"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1\") " pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" Nov 03 14:34:05.809180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.809148 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pth6\" (UniqueName: \"kubernetes.io/projected/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-kube-api-access-2pth6\") pod \"payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq\" (UID: \"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1\") " pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" Nov 03 14:34:05.888921 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.888833 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" Nov 03 14:34:05.945017 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:05.944990 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" Nov 03 14:34:06.192221 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:06.192196 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7"] Nov 03 14:34:06.193520 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:34:06.193496 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e3b5f4_8a45_4c0e_a370_f37e9da34eda.slice/crio-c1119de17a64b67e2bacc5cade2922dfdaf342d2b1d491148ccacf3a12c8b685 WatchSource:0}: Error finding container c1119de17a64b67e2bacc5cade2922dfdaf342d2b1d491148ccacf3a12c8b685: Status 404 returned error can't find the container with id c1119de17a64b67e2bacc5cade2922dfdaf342d2b1d491148ccacf3a12c8b685 Nov 03 14:34:06.269365 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:06.269331 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq"] Nov 03 14:34:06.781790 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:06.781750 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" event={"ID":"55e3b5f4-8a45-4c0e-a370-f37e9da34eda","Type":"ContainerStarted","Data":"4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573"} Nov 03 14:34:06.781992 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:06.781823 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" Nov 03 14:34:06.781992 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:06.781845 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" event={"ID":"55e3b5f4-8a45-4c0e-a370-f37e9da34eda","Type":"ContainerStarted","Data":"c1119de17a64b67e2bacc5cade2922dfdaf342d2b1d491148ccacf3a12c8b685"} Nov 03 14:34:06.783466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:06.783437 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" event={"ID":"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1","Type":"ContainerStarted","Data":"4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c"} Nov 03 14:34:06.783589 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:06.783478 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" event={"ID":"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1","Type":"ContainerStarted","Data":"e627bbc4003ed1b84b34ab885953695fec263f8a8f18ad47986103d2c24176d7"} Nov 03 14:34:06.783589 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:06.783547 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" Nov 03 14:34:06.828705 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:06.828650 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" podStartSLOduration=1.828623387 podStartE2EDuration="1.828623387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:06.826285371 +0000 UTC m=+1322.384921856" watchObservedRunningTime="2026-11-03 14:34:06.828623387 +0000 UTC m=+1322.387259872" Nov 03 14:34:06.871982 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:06.871935 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" podStartSLOduration=1.8719205730000001 podStartE2EDuration="1.871920573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:06.870211206 +0000 UTC m=+1322.428847690" watchObservedRunningTime="2026-11-03 14:34:06.871920573 +0000 UTC m=+1322.430557057" Nov 03 14:34:11.709652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:11.709508 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz"] Nov 03 14:34:11.710127 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:11.709809 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" containerID="cri-o://94959755c4be219a077ebe9e201a52ffcdeb5b9d9f0220db4a8a97a209dab966" gracePeriod=30 Nov 03 14:34:16.916394 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:16.916357 6389 generic.go:356] "Generic (PLEG): container finished" podID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerID="94959755c4be219a077ebe9e201a52ffcdeb5b9d9f0220db4a8a97a209dab966" exitCode=0 Nov 03 14:34:16.916800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:16.916437 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" event={"ID":"365dbedd-5d89-4991-ad96-ad20ebcfa1b5","Type":"ContainerDied","Data":"94959755c4be219a077ebe9e201a52ffcdeb5b9d9f0220db4a8a97a209dab966"} Nov 03 14:34:16.916800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:16.916501 6389 scope.go:122] "RemoveContainer" containerID="a5f85025f300040fe659b8216e5861351e282cab4c6f2d954a3a1b9b6d009b23" Nov 03 14:34:16.985990 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:16.985930 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:34:17.094106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.094073 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-envoy\") pod \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " Nov 03 14:34:17.094294 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.094117 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istiod-ca-cert\") pod \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " Nov 03 14:34:17.094294 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.094184 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-kube-api-access-ptqzn\" (UniqueName: \"kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-kube-api-access-ptqzn\") pod \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " Nov 03 14:34:17.094294 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.094252 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-credential-socket\") pod \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " Nov 03 14:34:17.094462 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.094325 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-certs\") pod \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " Nov 03 14:34:17.094462 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.094374 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-podinfo\") pod \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " Nov 03 14:34:17.094462 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.094438 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-socket\") pod \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " Nov 03 14:34:17.094656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.094495 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-data\") pod \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " Nov 03 14:34:17.094656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.094552 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-token\" (UniqueName: \"kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-token\") pod \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\" (UID: \"365dbedd-5d89-4991-ad96-ad20ebcfa1b5\") " Nov 03 14:34:17.094656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.094632 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istiod-ca-cert" pod "365dbedd-5d89-4991-ad96-ad20ebcfa1b5" (UID: "365dbedd-5d89-4991-ad96-ad20ebcfa1b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:34:17.094966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.094939 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-certs" pod "365dbedd-5d89-4991-ad96-ad20ebcfa1b5" (UID: "365dbedd-5d89-4991-ad96-ad20ebcfa1b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:34:17.095181 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.095134 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-socket" pod "365dbedd-5d89-4991-ad96-ad20ebcfa1b5" (UID: "365dbedd-5d89-4991-ad96-ad20ebcfa1b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:34:17.095398 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.095355 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:34:17.095398 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.095384 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:34:17.095398 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.095398 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:34:17.095635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.095349 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-data" pod "365dbedd-5d89-4991-ad96-ad20ebcfa1b5" (UID: "365dbedd-5d89-4991-ad96-ad20ebcfa1b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:34:17.095944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.095917 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-credential-socket" pod "365dbedd-5d89-4991-ad96-ad20ebcfa1b5" (UID: "365dbedd-5d89-4991-ad96-ad20ebcfa1b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:34:17.097364 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.097326 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-kube-api-access-ptqzn" pod "365dbedd-5d89-4991-ad96-ad20ebcfa1b5" (UID: "365dbedd-5d89-4991-ad96-ad20ebcfa1b5"). InnerVolumeSpecName "kube-api-access-ptqzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:34:17.097497 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.097468 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-podinfo" pod "365dbedd-5d89-4991-ad96-ad20ebcfa1b5" (UID: "365dbedd-5d89-4991-ad96-ad20ebcfa1b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:34:17.097587 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.097524 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-token" pod "365dbedd-5d89-4991-ad96-ad20ebcfa1b5" (UID: "365dbedd-5d89-4991-ad96-ad20ebcfa1b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:34:17.097587 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.097526 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-envoy" pod "365dbedd-5d89-4991-ad96-ad20ebcfa1b5" (UID: "365dbedd-5d89-4991-ad96-ad20ebcfa1b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:34:17.197162 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.197126 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:34:17.197162 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.197160 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:34:17.197162 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.197170 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:34:17.197417 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.197177 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:34:17.197417 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.197186 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:34:17.197417 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.197194 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptqzn\" (UniqueName: \"kubernetes.io/projected/365dbedd-5d89-4991-ad96-ad20ebcfa1b5-kube-api-access-ptqzn\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:34:17.793579 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.793551 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" Nov 03 14:34:17.793785 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.793741 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" Nov 03 14:34:17.928433 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.928403 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" event={"ID":"365dbedd-5d89-4991-ad96-ad20ebcfa1b5","Type":"ContainerDied","Data":"62bc6b25df6ca3ef51a9f42931b73a0046145f8a8771144fb69f29370198c6f9"} Nov 03 14:34:17.928918 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.928457 6389 scope.go:122] "RemoveContainer" containerID="94959755c4be219a077ebe9e201a52ffcdeb5b9d9f0220db4a8a97a209dab966" Nov 03 14:34:17.928918 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.928536 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz" Nov 03 14:34:17.988693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:17.988664 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz"] Nov 03 14:34:18.014195 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.013996 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-njjtz"] Nov 03 14:34:18.289301 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.289267 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8"] Nov 03 14:34:18.289895 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.289883 6389 cpu_manager.go:414] "containerMap: removing container" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" Nov 03 14:34:18.289954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.289897 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" Nov 03 14:34:18.289954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.289905 6389 cpu_manager.go:414] "containerMap: removing container" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" Nov 03 14:34:18.289954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.289910 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" Nov 03 14:34:18.290046 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.289999 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" Nov 03 14:34:18.290046 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.290007 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" containerName="istio-proxy" Nov 03 14:34:18.294486 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.294465 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" Nov 03 14:34:18.314243 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.314212 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8"] Nov 03 14:34:18.316927 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.316897 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-4a0349-dockercfg-qfkwn\"" Nov 03 14:34:18.317113 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.317088 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-4a0349\"" Nov 03 14:34:18.332266 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.332235 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7"] Nov 03 14:34:18.337707 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.337675 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" Nov 03 14:34:18.421636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.417051 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3495e145-fcb4-4338-8965-1581c04a6e4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8\" (UID: \"3495e145-fcb4-4338-8965-1581c04a6e4b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" Nov 03 14:34:18.421636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.417223 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngn9x\" (UniqueName: \"kubernetes.io/projected/3495e145-fcb4-4338-8965-1581c04a6e4b-kube-api-access-ngn9x\") pod \"payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8\" (UID: \"3495e145-fcb4-4338-8965-1581c04a6e4b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" Nov 03 14:34:18.421636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.417349 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8804c6c5-0c17-455e-82b9-89dee267575e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7\" (UID: \"8804c6c5-0c17-455e-82b9-89dee267575e\") " pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" Nov 03 14:34:18.421636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.417415 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqkhp\" (UniqueName: \"kubernetes.io/projected/8804c6c5-0c17-455e-82b9-89dee267575e-kube-api-access-sqkhp\") pod \"payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7\" (UID: \"8804c6c5-0c17-455e-82b9-89dee267575e\") " pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" Nov 03 14:34:18.509634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.492903 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7"] Nov 03 14:34:18.561702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.553405 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqkhp\" (UniqueName: \"kubernetes.io/projected/8804c6c5-0c17-455e-82b9-89dee267575e-kube-api-access-sqkhp\") pod \"payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7\" (UID: \"8804c6c5-0c17-455e-82b9-89dee267575e\") " pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" Nov 03 14:34:18.561702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.553546 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3495e145-fcb4-4338-8965-1581c04a6e4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8\" (UID: \"3495e145-fcb4-4338-8965-1581c04a6e4b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" Nov 03 14:34:18.561702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.553713 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngn9x\" (UniqueName: \"kubernetes.io/projected/3495e145-fcb4-4338-8965-1581c04a6e4b-kube-api-access-ngn9x\") pod \"payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8\" (UID: \"3495e145-fcb4-4338-8965-1581c04a6e4b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" Nov 03 14:34:18.561702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.553833 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8804c6c5-0c17-455e-82b9-89dee267575e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7\" (UID: \"8804c6c5-0c17-455e-82b9-89dee267575e\") " pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" Nov 03 14:34:18.561702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.554902 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8804c6c5-0c17-455e-82b9-89dee267575e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7\" (UID: \"8804c6c5-0c17-455e-82b9-89dee267575e\") " pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" Nov 03 14:34:18.561702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.555917 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3495e145-fcb4-4338-8965-1581c04a6e4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8\" (UID: \"3495e145-fcb4-4338-8965-1581c04a6e4b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" Nov 03 14:34:18.583701 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.581042 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqkhp\" (UniqueName: \"kubernetes.io/projected/8804c6c5-0c17-455e-82b9-89dee267575e-kube-api-access-sqkhp\") pod \"payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7\" (UID: \"8804c6c5-0c17-455e-82b9-89dee267575e\") " pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" Nov 03 14:34:18.590929 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.590896 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngn9x\" (UniqueName: \"kubernetes.io/projected/3495e145-fcb4-4338-8965-1581c04a6e4b-kube-api-access-ngn9x\") pod \"payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8\" (UID: \"3495e145-fcb4-4338-8965-1581c04a6e4b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" Nov 03 14:34:18.608697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.608670 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" Nov 03 14:34:18.620209 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.620170 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365dbedd-5d89-4991-ad96-ad20ebcfa1b5" path="/var/lib/kubelet/pods/365dbedd-5d89-4991-ad96-ad20ebcfa1b5/volumes" Nov 03 14:34:18.708035 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.707762 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" Nov 03 14:34:18.924795 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.924766 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8"] Nov 03 14:34:18.925598 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:34:18.925572 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3495e145_fcb4_4338_8965_1581c04a6e4b.slice/crio-c98e052b72ee32859dff5c3fa75a3ba11b29875e8c48da20c18ce7835e2a2276 WatchSource:0}: Error finding container c98e052b72ee32859dff5c3fa75a3ba11b29875e8c48da20c18ce7835e2a2276: Status 404 returned error can't find the container with id c98e052b72ee32859dff5c3fa75a3ba11b29875e8c48da20c18ce7835e2a2276 Nov 03 14:34:18.941325 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:18.941296 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" event={"ID":"3495e145-fcb4-4338-8965-1581c04a6e4b","Type":"ContainerStarted","Data":"c98e052b72ee32859dff5c3fa75a3ba11b29875e8c48da20c18ce7835e2a2276"} Nov 03 14:34:19.031629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:19.031584 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7"] Nov 03 14:34:19.032716 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:34:19.032684 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8804c6c5_0c17_455e_82b9_89dee267575e.slice/crio-12594ba3732e7138fc86db3cba05481b731620cfeabc658ad655221cbc76e239 WatchSource:0}: Error finding container 12594ba3732e7138fc86db3cba05481b731620cfeabc658ad655221cbc76e239: Status 404 returned error can't find the container with id 12594ba3732e7138fc86db3cba05481b731620cfeabc658ad655221cbc76e239 Nov 03 14:34:19.950040 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:19.950004 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" event={"ID":"3495e145-fcb4-4338-8965-1581c04a6e4b","Type":"ContainerStarted","Data":"84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad"} Nov 03 14:34:19.950502 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:19.950069 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" Nov 03 14:34:19.951732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:19.951706 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" event={"ID":"8804c6c5-0c17-455e-82b9-89dee267575e","Type":"ContainerStarted","Data":"9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c"} Nov 03 14:34:19.951870 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:19.951750 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" event={"ID":"8804c6c5-0c17-455e-82b9-89dee267575e","Type":"ContainerStarted","Data":"12594ba3732e7138fc86db3cba05481b731620cfeabc658ad655221cbc76e239"} Nov 03 14:34:19.951870 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:19.951794 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" Nov 03 14:34:19.992393 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:19.992346 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" podStartSLOduration=1.992333031 podStartE2EDuration="1.992333031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:19.991696262 +0000 UTC m=+1335.550332751" watchObservedRunningTime="2026-11-03 14:34:19.992333031 +0000 UTC m=+1335.550969515" Nov 03 14:34:20.036950 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:20.036898 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" podStartSLOduration=2.036885012 podStartE2EDuration="2.036885012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:20.035098305 +0000 UTC m=+1335.593734789" watchObservedRunningTime="2026-11-03 14:34:20.036885012 +0000 UTC m=+1335.595521496" Nov 03 14:34:28.401347 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:34:28.401308 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:34:30.977602 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:30.977574 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" Nov 03 14:34:30.983493 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:30.983471 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" Nov 03 14:34:34.275134 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.275096 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd"] Nov 03 14:34:34.280039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.280018 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:34.310320 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.310285 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-4a0349-serving-cert\"" Nov 03 14:34:34.315083 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.315051 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd"] Nov 03 14:34:34.343349 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.343316 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhrr\" (UniqueName: \"kubernetes.io/projected/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-kube-api-access-bjhrr\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:34.343530 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.343505 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-openshift-service-ca\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:34.343634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.343597 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-metrics-certs\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:34.343774 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.343753 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-maas-api-tls\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:34.444870 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.444832 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhrr\" (UniqueName: \"kubernetes.io/projected/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-kube-api-access-bjhrr\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:34.445042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.444919 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-openshift-service-ca\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:34.445042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.444958 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-metrics-certs\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:34.445042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.445031 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-maas-api-tls\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:34.445174 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:34:34.445159 6389 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-serving-cert: secret "maas-api-e2e-explicit-4a0349-serving-cert" not found Nov 03 14:34:34.445241 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:34:34.445231 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-maas-api-tls podName:ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a nodeName:}" failed. No retries permitted until 2026-11-03 14:34:34.945211122 +0000 UTC m=+1350.503847585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-maas-api-tls") pod "maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" (UID: "ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a") : secret "maas-api-e2e-explicit-4a0349-serving-cert" not found Nov 03 14:34:34.445574 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.445555 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-openshift-service-ca\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:34.447620 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.447580 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-metrics-certs\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:34.470444 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.470412 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhrr\" (UniqueName: \"kubernetes.io/projected/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-kube-api-access-bjhrr\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:34.974793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:34.974752 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-maas-api-tls\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:35.015634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:35.014405 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-maas-api-tls\") pod \"maas-api-e2e-explicit-4a0349-7d8978658d-c92nd\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:35.195083 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:35.195044 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:35.527751 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:35.527719 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd"] Nov 03 14:34:35.528305 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:34:35.528271 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded90a8d6_c348_4e7c_add4_c5bbb7d91e0a.slice/crio-a8c7d3d587f55341dfaa6a0a5f39fad822b508df4126a1fb5f5cbd11463cb741 WatchSource:0}: Error finding container a8c7d3d587f55341dfaa6a0a5f39fad822b508df4126a1fb5f5cbd11463cb741: Status 404 returned error can't find the container with id a8c7d3d587f55341dfaa6a0a5f39fad822b508df4126a1fb5f5cbd11463cb741 Nov 03 14:34:36.103409 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.103376 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" event={"ID":"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a","Type":"ContainerStarted","Data":"2868d259b9a2cfbccdc8c58fd8223eaa2155ca1f81fd6a0156238b2926dbaae9"} Nov 03 14:34:36.103628 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.103444 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:36.103628 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.103467 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" event={"ID":"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a","Type":"ContainerStarted","Data":"a8c7d3d587f55341dfaa6a0a5f39fad822b508df4126a1fb5f5cbd11463cb741"} Nov 03 14:34:36.148146 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.148096 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" podStartSLOduration=1.748991999 podStartE2EDuration="2.148081475s" totalImagesPullingTime="399.089476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:36.146197282 +0000 UTC m=+1351.704833765" watchObservedRunningTime="2026-11-03 14:34:36.148081475 +0000 UTC m=+1351.706717960" Nov 03 14:34:36.387814 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.387724 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw"] Nov 03 14:34:36.392371 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.392353 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.414863 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.414832 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw"] Nov 03 14:34:36.429481 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.429444 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.429481 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.429492 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnzg\" (UniqueName: \"kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-kube-api-access-djnzg\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.429864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.429542 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.429864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.429583 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.429864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.429656 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.429864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.429744 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.429864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.429786 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.429864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.429809 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.429864 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.429843 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.530665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.530602 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.530665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.530668 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.531190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.530693 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.531190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.530771 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.531190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.530786 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnzg\" (UniqueName: \"kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-kube-api-access-djnzg\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.531190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.530823 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.531190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.530867 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.531190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.530925 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.531190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.531023 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.531190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.531058 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.531190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.531110 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.531190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.531186 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.531519 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.531346 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.531715 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.531698 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.533225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.533204 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.533569 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.533553 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.552718 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.552686 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnzg\" (UniqueName: \"kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-kube-api-access-djnzg\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.553003 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.552984 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qbxzw\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:36.710873 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:36.710825 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:37.097018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:37.096995 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw"] Nov 03 14:34:37.097414 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:34:37.097392 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1fd640_11de_42e6_8ab7_19ced0bcbb05.slice/crio-3a828edb64d2b3074af3ecdb3bd7651ef793387bdbb2a837039ad8a95e10cb49 WatchSource:0}: Error finding container 3a828edb64d2b3074af3ecdb3bd7651ef793387bdbb2a837039ad8a95e10cb49: Status 404 returned error can't find the container with id 3a828edb64d2b3074af3ecdb3bd7651ef793387bdbb2a837039ad8a95e10cb49 Nov 03 14:34:37.099968 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:37.099938 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:34:37.100041 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:37.100007 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:34:37.100081 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:37.100043 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:34:37.111381 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:37.111353 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" event={"ID":"4c1fd640-11de-42e6-8ab7-19ced0bcbb05","Type":"ContainerStarted","Data":"3a828edb64d2b3074af3ecdb3bd7651ef793387bdbb2a837039ad8a95e10cb49"} Nov 03 14:34:38.142632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:38.142446 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" event={"ID":"4c1fd640-11de-42e6-8ab7-19ced0bcbb05","Type":"ContainerStarted","Data":"439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac"} Nov 03 14:34:38.217981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:38.217932 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" podStartSLOduration=2.217918322 podStartE2EDuration="2.217918322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:38.202348757 +0000 UTC m=+1353.760985243" watchObservedRunningTime="2026-11-03 14:34:38.217918322 +0000 UTC m=+1353.776554806" Nov 03 14:34:38.711148 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:38.711120 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:38.712865 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:38.712841 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:34:38.712992 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:38.712885 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 14:34:39.711338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:39.711305 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:34:39.711771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:39.711362 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 14:34:40.737633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:40.736296 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:34:40.737633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:40.736360 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 14:34:41.711160 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:41.711125 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:34:41.711355 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:41.711192 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 14:34:42.117656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:42.117548 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:34:42.711833 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:42.711801 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:34:42.712011 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:42.711860 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 14:34:43.711886 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:43.711808 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:34:43.711886 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:43.711858 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 14:34:44.711239 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:44.711205 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:34:44.711420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:44.711267 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 14:34:45.021945 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.021847 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m"] Nov 03 14:34:45.028697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.028671 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.057289 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.057257 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2e8614-openshift-default-dockercfg-tbwqd\"" Nov 03 14:34:45.061527 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.061503 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m"] Nov 03 14:34:45.094337 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.094286 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-data\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.094754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.094731 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndl5l\" (UniqueName: \"kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-kube-api-access-ndl5l\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.095046 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.095027 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-certs\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.095286 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.095270 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-token\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.095509 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.095491 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-podinfo\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.095736 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.095713 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09311533-6310-4062-b1a2-1167c4e4bbc2-istiod-ca-cert\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.095903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.095888 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-socket\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.096097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.096081 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-envoy\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.096302 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.096287 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-credential-socket\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.243516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.243480 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-certs\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.255854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.255815 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-token\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.264701 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.248250 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-certs\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.281804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.277574 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-podinfo\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.281804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.277843 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09311533-6310-4062-b1a2-1167c4e4bbc2-istiod-ca-cert\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.281804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.277874 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-socket\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.281804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.278177 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-envoy\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.281804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.278475 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-credential-socket\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.281804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.278657 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-data\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.281804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.278839 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndl5l\" (UniqueName: \"kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-kube-api-access-ndl5l\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.303182 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.303094 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09311533-6310-4062-b1a2-1167c4e4bbc2-istiod-ca-cert\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.303859 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.303804 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-token\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.304188 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.304130 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-credential-socket\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.304435 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.304416 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-data\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.305036 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.305000 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-socket\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.321648 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.320688 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-envoy\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.342684 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.342648 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-podinfo\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.360793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.360746 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndl5l\" (UniqueName: \"kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-kube-api-access-ndl5l\") pod \"e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.643596 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.643502 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:45.711251 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.711217 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:34:45.711423 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:45.711285 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 14:34:46.012728 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:46.012700 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m"] Nov 03 14:34:46.015050 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:34:46.015022 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09311533_6310_4062_b1a2_1167c4e4bbc2.slice/crio-ef94f2f1732bafd931e0997893c58c68fa509115a644399f90b85d0347747678 WatchSource:0}: Error finding container ef94f2f1732bafd931e0997893c58c68fa509115a644399f90b85d0347747678: Status 404 returned error can't find the container with id ef94f2f1732bafd931e0997893c58c68fa509115a644399f90b85d0347747678 Nov 03 14:34:46.018321 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:46.018284 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:34:46.018399 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:46.018373 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:34:46.018443 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:46.018422 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:34:46.301703 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:46.300557 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" event={"ID":"09311533-6310-4062-b1a2-1167c4e4bbc2","Type":"ContainerStarted","Data":"21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9"} Nov 03 14:34:46.301703 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:46.300637 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" event={"ID":"09311533-6310-4062-b1a2-1167c4e4bbc2","Type":"ContainerStarted","Data":"ef94f2f1732bafd931e0997893c58c68fa509115a644399f90b85d0347747678"} Nov 03 14:34:46.352242 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:46.352187 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" podStartSLOduration=2.352173438 podStartE2EDuration="2.352173438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:46.350892687 +0000 UTC m=+1361.909529172" watchObservedRunningTime="2026-11-03 14:34:46.352173438 +0000 UTC m=+1361.910809921" Nov 03 14:34:46.644417 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:46.644315 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:47.069724 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.069690 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:47.070078 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.070057 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:47.071484 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.071465 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:34:47.091911 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.091885 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl"] Nov 03 14:34:47.096341 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.096321 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.123119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.123093 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl"] Nov 03 14:34:47.221144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.220181 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.221144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.220231 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.221144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.220440 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.221144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.220722 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.221144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.220750 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.221144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.220809 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.221144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.220833 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a413656c-557f-4262-b40c-a0cf1e193ac6-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.221144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.220915 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.221144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.220985 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxpsf\" (UniqueName: \"kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-kube-api-access-kxpsf\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.325842 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.325751 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.326221 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.325886 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.326221 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.325915 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.326221 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.325950 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.326221 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.325971 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a413656c-557f-4262-b40c-a0cf1e193ac6-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.326221 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.326022 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.326221 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.326058 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxpsf\" (UniqueName: \"kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-kube-api-access-kxpsf\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.326488 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.326427 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.326488 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.326451 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.326939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.326777 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.326939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.326887 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.326939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.326921 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.327196 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.327057 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a413656c-557f-4262-b40c-a0cf1e193ac6-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.327256 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.327206 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.328960 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.328936 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.331231 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.331206 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.348915 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.348885 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxpsf\" (UniqueName: \"kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-kube-api-access-kxpsf\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.349299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.349279 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.412834 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.412799 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:47.662237 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.662122 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:47.737024 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.737000 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl"] Nov 03 14:34:47.738032 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:34:47.738008 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda413656c_557f_4262_b40c_a0cf1e193ac6.slice/crio-6b525bee9be679db26fa912f7c57f88b38308148ec5335773e468383ee832456 WatchSource:0}: Error finding container 6b525bee9be679db26fa912f7c57f88b38308148ec5335773e468383ee832456: Status 404 returned error can't find the container with id 6b525bee9be679db26fa912f7c57f88b38308148ec5335773e468383ee832456 Nov 03 14:34:47.741007 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.740976 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:34:47.741101 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.741033 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:34:47.741101 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:47.741061 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:34:48.337011 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:48.336974 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" event={"ID":"a413656c-557f-4262-b40c-a0cf1e193ac6","Type":"ContainerStarted","Data":"19022acb1d19eb1fe2c878bc10a0dcd231f68bf4e745d5701c89efcaf1a93087"} Nov 03 14:34:48.337493 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:48.337081 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" event={"ID":"a413656c-557f-4262-b40c-a0cf1e193ac6","Type":"ContainerStarted","Data":"6b525bee9be679db26fa912f7c57f88b38308148ec5335773e468383ee832456"} Nov 03 14:34:48.337835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:48.337814 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:48.338786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:48.338766 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:34:48.389447 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:48.389380 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" podStartSLOduration=1.389362738 podStartE2EDuration="1.389362738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:48.387127608 +0000 UTC m=+1363.945764100" watchObservedRunningTime="2026-11-03 14:34:48.389362738 +0000 UTC m=+1363.947999223" Nov 03 14:34:48.413504 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:48.413466 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:48.414874 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:48.414844 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:34:48.414993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:48.414898 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" podUID="a413656c-557f-4262-b40c-a0cf1e193ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 14:34:49.413720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:49.413689 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:34:49.414187 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:49.413739 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" podUID="a413656c-557f-4262-b40c-a0cf1e193ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 14:34:50.414034 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:50.414000 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:34:50.414441 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:50.414052 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" podUID="a413656c-557f-4262-b40c-a0cf1e193ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 14:34:51.413649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:51.413577 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:34:51.413823 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:51.413668 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" podUID="a413656c-557f-4262-b40c-a0cf1e193ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 14:34:52.413793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:52.413761 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:34:52.414394 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:52.414364 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" podUID="a413656c-557f-4262-b40c-a0cf1e193ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 14:34:53.414167 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:53.414130 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:34:53.414584 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:53.414239 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" podUID="a413656c-557f-4262-b40c-a0cf1e193ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 14:34:54.413293 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:54.413260 6389 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:34:54.413461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:54.413328 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" podUID="a413656c-557f-4262-b40c-a0cf1e193ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 14:34:55.817430 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:55.817390 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5"] Nov 03 14:34:56.367056 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:56.367023 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:56.368978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:56.368956 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:56.372148 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:34:56.372126 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:34:58.591747 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:34:58.591713 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:35:00.818588 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:00.818527 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" podUID="c31f97c3-0499-4900-bd83-9c0cfe4a3c5c" containerName="payload-processing" containerID="cri-o://9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81" gracePeriod=25 Nov 03 14:35:01.303034 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.303010 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:35:01.450096 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.450021 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-plugins-config-volume\") pod \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " Nov 03 14:35:01.450096 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.450064 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-kube-api-access-2hz8k\" (UniqueName: \"kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-kube-api-access-2hz8k\") pod \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " Nov 03 14:35:01.450334 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.450155 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-routing-overlay-volume\") pod \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " Nov 03 14:35:01.450334 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.450182 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-provider-credentials\") pod \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\" (UID: \"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c\") " Nov 03 14:35:01.450545 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.450517 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-plugins-config-volume" pod "c31f97c3-0499-4900-bd83-9c0cfe4a3c5c" (UID: "c31f97c3-0499-4900-bd83-9c0cfe4a3c5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:35:01.450680 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.450642 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-routing-overlay-volume" pod "c31f97c3-0499-4900-bd83-9c0cfe4a3c5c" (UID: "c31f97c3-0499-4900-bd83-9c0cfe4a3c5c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:35:01.450757 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.450740 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:35:01.452375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.452357 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-provider-credentials" pod "c31f97c3-0499-4900-bd83-9c0cfe4a3c5c" (UID: "c31f97c3-0499-4900-bd83-9c0cfe4a3c5c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:35:01.452438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.452398 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-kube-api-access-2hz8k" pod "c31f97c3-0499-4900-bd83-9c0cfe4a3c5c" (UID: "c31f97c3-0499-4900-bd83-9c0cfe4a3c5c"). InnerVolumeSpecName "kube-api-access-2hz8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:35:01.563634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.560215 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:35:01.563634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.560255 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:35:01.563634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.560270 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hz8k\" (UniqueName: \"kubernetes.io/projected/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c-kube-api-access-2hz8k\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:35:01.610631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.607334 6389 generic.go:356] "Generic (PLEG): container finished" podID="c31f97c3-0499-4900-bd83-9c0cfe4a3c5c" containerID="9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81" exitCode=0 Nov 03 14:35:01.610631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.607414 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" event={"ID":"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c","Type":"ContainerDied","Data":"9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81"} Nov 03 14:35:01.610631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.607492 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" event={"ID":"c31f97c3-0499-4900-bd83-9c0cfe4a3c5c","Type":"ContainerDied","Data":"d29f97264cb6239f52d20c1aeee9c25c167df9afeffef3306f6c133a9bb60259"} Nov 03 14:35:01.610631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.607525 6389 scope.go:122] "RemoveContainer" containerID="9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81" Nov 03 14:35:01.610631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.607768 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5" Nov 03 14:35:01.693589 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.693478 6389 scope.go:122] "RemoveContainer" containerID="9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81" Nov 03 14:35:01.694076 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:35:01.694041 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81\": container with ID starting with 9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81 not found: ID does not exist" containerID="9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81" Nov 03 14:35:01.694247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.694216 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81"} err="failed to get container status \"9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81\": rpc error: code = NotFound desc = could not find container \"9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81\": container with ID starting with 9fe540c35ea2b80e6eeb21befc407819bf101ce03c0d512a3c41495b24575c81 not found: ID does not exist" Nov 03 14:35:01.728202 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.728116 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5"] Nov 03 14:35:01.731185 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:01.731152 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5f45tg9k5"] Nov 03 14:35:02.620384 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:02.620350 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31f97c3-0499-4900-bd83-9c0cfe4a3c5c" path="/var/lib/kubelet/pods/c31f97c3-0499-4900-bd83-9c0cfe4a3c5c/volumes" Nov 03 14:35:28.378262 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:35:28.378223 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:35:33.042168 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:33.042140 6389 generic.go:356] "Generic (PLEG): container finished" podID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerID="21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9" exitCode=0 Nov 03 14:35:33.042713 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:33.042217 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" event={"ID":"09311533-6310-4062-b1a2-1167c4e4bbc2","Type":"ContainerDied","Data":"21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9"} Nov 03 14:35:33.043301 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:33.043279 6389 scope.go:122] "RemoveContainer" containerID="21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9" Nov 03 14:35:33.045121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:33.045090 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:35:33.045216 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:33.045179 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:35:33.045279 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:33.045229 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:35:33.360762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:33.347827 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:35:34.064989 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:34.064946 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" event={"ID":"09311533-6310-4062-b1a2-1167c4e4bbc2","Type":"ContainerStarted","Data":"74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d"} Nov 03 14:35:34.649181 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:34.649148 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:35:34.654717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:34.654687 6389 patch_prober.go:29] interesting pod/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 14:35:34.654979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:34.654948 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 14:35:35.648640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:35.645238 6389 patch_prober.go:29] interesting pod/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 14:35:35.677747 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:35.677701 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 14:35:36.652635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:36.652444 6389 patch_prober.go:29] interesting pod/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 14:35:36.652635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:36.652511 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 14:35:37.646643 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:37.646010 6389 patch_prober.go:29] interesting pod/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 14:35:37.646643 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:37.646082 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 14:35:38.645972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:38.645933 6389 patch_prober.go:29] interesting pod/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 14:35:38.646585 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:38.645993 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 14:35:39.645256 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:39.645229 6389 patch_prober.go:29] interesting pod/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 14:35:39.645496 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:39.645468 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 14:35:40.649018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:40.648988 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:35:40.649479 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:40.649322 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:35:40.650140 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:40.650120 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:35:44.356213 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:44.356178 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq"] Nov 03 14:35:44.391591 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:44.391557 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7"] Nov 03 14:35:49.390750 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:49.390693 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" podUID="a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1" containerName="payload-processing" containerID="cri-o://4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c" gracePeriod=25 Nov 03 14:35:49.394448 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:49.392245 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" podUID="55e3b5f4-8a45-4c0e-a370-f37e9da34eda" containerName="payload-pre-processing" containerID="cri-o://4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573" gracePeriod=25 Nov 03 14:35:50.215821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.215416 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" Nov 03 14:35:50.378360 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.378338 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" Nov 03 14:35:50.403973 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.403835 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-plugins-config-volume\") pod \"55e3b5f4-8a45-4c0e-a370-f37e9da34eda\" (UID: \"55e3b5f4-8a45-4c0e-a370-f37e9da34eda\") " Nov 03 14:35:50.404461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.404194 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-kube-api-access-r8cml\" (UniqueName: \"kubernetes.io/projected/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-kube-api-access-r8cml\") pod \"55e3b5f4-8a45-4c0e-a370-f37e9da34eda\" (UID: \"55e3b5f4-8a45-4c0e-a370-f37e9da34eda\") " Nov 03 14:35:50.409964 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.406198 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-plugins-config-volume" pod "55e3b5f4-8a45-4c0e-a370-f37e9da34eda" (UID: "55e3b5f4-8a45-4c0e-a370-f37e9da34eda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:35:50.409964 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.407452 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-kube-api-access-r8cml" pod "55e3b5f4-8a45-4c0e-a370-f37e9da34eda" (UID: "55e3b5f4-8a45-4c0e-a370-f37e9da34eda"). InnerVolumeSpecName "kube-api-access-r8cml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:35:50.415816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.411993 6389 generic.go:356] "Generic (PLEG): container finished" podID="55e3b5f4-8a45-4c0e-a370-f37e9da34eda" containerID="4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573" exitCode=0 Nov 03 14:35:50.415816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.412059 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" Nov 03 14:35:50.415816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.412129 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" event={"ID":"55e3b5f4-8a45-4c0e-a370-f37e9da34eda","Type":"ContainerDied","Data":"4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573"} Nov 03 14:35:50.415816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.412205 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7" event={"ID":"55e3b5f4-8a45-4c0e-a370-f37e9da34eda","Type":"ContainerDied","Data":"c1119de17a64b67e2bacc5cade2922dfdaf342d2b1d491148ccacf3a12c8b685"} Nov 03 14:35:50.415816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.412235 6389 scope.go:122] "RemoveContainer" containerID="4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573" Nov 03 14:35:50.419982 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.419956 6389 generic.go:356] "Generic (PLEG): container finished" podID="a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1" containerID="4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c" exitCode=0 Nov 03 14:35:50.420203 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.420044 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" Nov 03 14:35:50.420375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.420065 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" event={"ID":"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1","Type":"ContainerDied","Data":"4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c"} Nov 03 14:35:50.420524 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.420506 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq" event={"ID":"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1","Type":"ContainerDied","Data":"e627bbc4003ed1b84b34ab885953695fec263f8a8f18ad47986103d2c24176d7"} Nov 03 14:35:50.443230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.443204 6389 scope.go:122] "RemoveContainer" containerID="4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573" Nov 03 14:35:50.444200 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:35:50.444171 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573\": container with ID starting with 4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573 not found: ID does not exist" containerID="4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573" Nov 03 14:35:50.444345 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.444318 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573"} err="failed to get container status \"4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573\": rpc error: code = NotFound desc = could not find container \"4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573\": container with ID starting with 4f3309328700b4740fa0007ef65aa2fe2e29f459cbafc2aae8a7944825f0a573 not found: ID does not exist" Nov 03 14:35:50.444462 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.444449 6389 scope.go:122] "RemoveContainer" containerID="4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c" Nov 03 14:35:50.459384 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.459354 6389 scope.go:122] "RemoveContainer" containerID="4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c" Nov 03 14:35:50.459869 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:35:50.459841 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c\": container with ID starting with 4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c not found: ID does not exist" containerID="4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c" Nov 03 14:35:50.459982 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.459882 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c"} err="failed to get container status \"4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c\": rpc error: code = NotFound desc = could not find container \"4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c\": container with ID starting with 4102eedf89b347d07d93a081367d922bb98b9386bd51a2ef6106ae817bc2b71c not found: ID does not exist" Nov 03 14:35:50.507625 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.507573 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-plugins-config-volume\") pod \"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1\" (UID: \"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1\") " Nov 03 14:35:50.507810 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.507651 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-kube-api-access-2pth6\" (UniqueName: \"kubernetes.io/projected/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-kube-api-access-2pth6\") pod \"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1\" (UID: \"a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1\") " Nov 03 14:35:50.508053 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.508030 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-plugins-config-volume" pod "a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1" (UID: "a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:35:50.508166 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.508092 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7"] Nov 03 14:35:50.508242 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.508225 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:35:50.508279 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.508247 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8cml\" (UniqueName: \"kubernetes.io/projected/55e3b5f4-8a45-4c0e-a370-f37e9da34eda-kube-api-access-r8cml\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:35:50.508279 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.508258 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:35:50.510024 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.510002 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-kube-api-access-2pth6" pod "a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1" (UID: "a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1"). InnerVolumeSpecName "kube-api-access-2pth6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:35:50.516262 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.516238 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cb587c-5ccf5b4447-m4hn7"] Nov 03 14:35:50.610031 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.609701 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pth6\" (UniqueName: \"kubernetes.io/projected/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1-kube-api-access-2pth6\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:35:50.625225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.625187 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e3b5f4-8a45-4c0e-a370-f37e9da34eda" path="/var/lib/kubelet/pods/55e3b5f4-8a45-4c0e-a370-f37e9da34eda/volumes" Nov 03 14:35:50.763226 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.763189 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq"] Nov 03 14:35:50.770289 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:50.770264 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cb587c-749d8bd9d7-7mrwq"] Nov 03 14:35:52.619788 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:35:52.619755 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1" path="/var/lib/kubelet/pods/a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1/volumes" Nov 03 14:35:58.375017 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:35:58.374976 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:36:05.742939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:05.742902 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7"] Nov 03 14:36:05.819644 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:05.819582 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8"] Nov 03 14:36:10.768907 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:10.768867 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" podUID="8804c6c5-0c17-455e-82b9-89dee267575e" containerName="payload-processing" containerID="cri-o://9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c" gracePeriod=25 Nov 03 14:36:10.821678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:10.821580 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" podUID="3495e145-fcb4-4338-8965-1581c04a6e4b" containerName="payload-pre-processing" containerID="cri-o://84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad" gracePeriod=25 Nov 03 14:36:10.976489 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:36:10.976446 6389 logging.go:55] [core] [Channel #2128 SubChannel #2129]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.112:50052", ServerName: "10.134.0.112:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.112:50052: connect: connection refused" Nov 03 14:36:10.981826 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:36:10.981801 6389 logging.go:55] [core] [Channel #2130 SubChannel #2131]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.113:50052", ServerName: "10.134.0.113:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.113:50052: connect: connection refused" Nov 03 14:36:11.096519 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.096491 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" Nov 03 14:36:11.128922 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.128898 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" Nov 03 14:36:11.131100 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.131078 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8804c6c5-0c17-455e-82b9-89dee267575e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8804c6c5-0c17-455e-82b9-89dee267575e-plugins-config-volume\") pod \"8804c6c5-0c17-455e-82b9-89dee267575e\" (UID: \"8804c6c5-0c17-455e-82b9-89dee267575e\") " Nov 03 14:36:11.131375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.131353 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8804c6c5-0c17-455e-82b9-89dee267575e-kube-api-access-sqkhp\" (UniqueName: \"kubernetes.io/projected/8804c6c5-0c17-455e-82b9-89dee267575e-kube-api-access-sqkhp\") pod \"8804c6c5-0c17-455e-82b9-89dee267575e\" (UID: \"8804c6c5-0c17-455e-82b9-89dee267575e\") " Nov 03 14:36:11.131482 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.131458 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8804c6c5-0c17-455e-82b9-89dee267575e-plugins-config-volume" pod "8804c6c5-0c17-455e-82b9-89dee267575e" (UID: "8804c6c5-0c17-455e-82b9-89dee267575e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:36:11.131835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.131819 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8804c6c5-0c17-455e-82b9-89dee267575e-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:11.133492 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.133469 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8804c6c5-0c17-455e-82b9-89dee267575e-kube-api-access-sqkhp" pod "8804c6c5-0c17-455e-82b9-89dee267575e" (UID: "8804c6c5-0c17-455e-82b9-89dee267575e"). InnerVolumeSpecName "kube-api-access-sqkhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:11.233203 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.233169 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3495e145-fcb4-4338-8965-1581c04a6e4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3495e145-fcb4-4338-8965-1581c04a6e4b-plugins-config-volume\") pod \"3495e145-fcb4-4338-8965-1581c04a6e4b\" (UID: \"3495e145-fcb4-4338-8965-1581c04a6e4b\") " Nov 03 14:36:11.233390 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.233213 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3495e145-fcb4-4338-8965-1581c04a6e4b-kube-api-access-ngn9x\" (UniqueName: \"kubernetes.io/projected/3495e145-fcb4-4338-8965-1581c04a6e4b-kube-api-access-ngn9x\") pod \"3495e145-fcb4-4338-8965-1581c04a6e4b\" (UID: \"3495e145-fcb4-4338-8965-1581c04a6e4b\") " Nov 03 14:36:11.233664 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.233639 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3495e145-fcb4-4338-8965-1581c04a6e4b-plugins-config-volume" pod "3495e145-fcb4-4338-8965-1581c04a6e4b" (UID: "3495e145-fcb4-4338-8965-1581c04a6e4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:36:11.233845 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.233832 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqkhp\" (UniqueName: \"kubernetes.io/projected/8804c6c5-0c17-455e-82b9-89dee267575e-kube-api-access-sqkhp\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:11.233895 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.233848 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3495e145-fcb4-4338-8965-1581c04a6e4b-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:11.235508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.235488 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3495e145-fcb4-4338-8965-1581c04a6e4b-kube-api-access-ngn9x" pod "3495e145-fcb4-4338-8965-1581c04a6e4b" (UID: "3495e145-fcb4-4338-8965-1581c04a6e4b"). InnerVolumeSpecName "kube-api-access-ngn9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:11.345690 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.343775 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngn9x\" (UniqueName: \"kubernetes.io/projected/3495e145-fcb4-4338-8965-1581c04a6e4b-kube-api-access-ngn9x\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:11.726039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.726008 6389 generic.go:356] "Generic (PLEG): container finished" podID="3495e145-fcb4-4338-8965-1581c04a6e4b" containerID="84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad" exitCode=0 Nov 03 14:36:11.726245 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.726070 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" Nov 03 14:36:11.726245 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.726082 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" event={"ID":"3495e145-fcb4-4338-8965-1581c04a6e4b","Type":"ContainerDied","Data":"84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad"} Nov 03 14:36:11.726245 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.726147 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" event={"ID":"3495e145-fcb4-4338-8965-1581c04a6e4b","Type":"ContainerDied","Data":"c98e052b72ee32859dff5c3fa75a3ba11b29875e8c48da20c18ce7835e2a2276"} Nov 03 14:36:11.726245 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.726182 6389 scope.go:122] "RemoveContainer" containerID="84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad" Nov 03 14:36:11.727744 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.727713 6389 generic.go:356] "Generic (PLEG): container finished" podID="8804c6c5-0c17-455e-82b9-89dee267575e" containerID="9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c" exitCode=0 Nov 03 14:36:11.727894 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.727758 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" Nov 03 14:36:11.727894 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.727796 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" event={"ID":"8804c6c5-0c17-455e-82b9-89dee267575e","Type":"ContainerDied","Data":"9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c"} Nov 03 14:36:11.727894 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.727845 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" event={"ID":"8804c6c5-0c17-455e-82b9-89dee267575e","Type":"ContainerDied","Data":"12594ba3732e7138fc86db3cba05481b731620cfeabc658ad655221cbc76e239"} Nov 03 14:36:11.736834 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.736820 6389 scope.go:122] "RemoveContainer" containerID="84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad" Nov 03 14:36:11.737158 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:36:11.737129 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad\": container with ID starting with 84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad not found: ID does not exist" containerID="84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad" Nov 03 14:36:11.737271 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.737163 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad"} err="failed to get container status \"84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad\": rpc error: code = NotFound desc = could not find container \"84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad\": container with ID starting with 84ec9ba58139a9f0ad9e8123664d1967674ebec2d961ccd4e412c04ce39298ad not found: ID does not exist" Nov 03 14:36:11.737271 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.737179 6389 scope.go:122] "RemoveContainer" containerID="9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c" Nov 03 14:36:11.747483 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.747468 6389 scope.go:122] "RemoveContainer" containerID="9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c" Nov 03 14:36:11.747760 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:36:11.747743 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c\": container with ID starting with 9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c not found: ID does not exist" containerID="9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c" Nov 03 14:36:11.747825 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.747767 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c"} err="failed to get container status \"9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c\": rpc error: code = NotFound desc = could not find container \"9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c\": container with ID starting with 9c3a76106287dd071b75c942550ff3f4010536cfc1f34c9f4107e57acdc3542c not found: ID does not exist" Nov 03 14:36:11.783850 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.783810 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8"] Nov 03 14:36:11.794266 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.794241 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8"] Nov 03 14:36:11.825803 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.825771 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7"] Nov 03 14:36:11.831749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.831723 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7"] Nov 03 14:36:11.990692 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.985973 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a0349-595768c78c-6wfx8" podUID="3495e145-fcb4-4338-8965-1581c04a6e4b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.112:50052\" within 1s: context deadline exceeded" Nov 03 14:36:11.990692 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:36:11.986391 6389 logging.go:55] [core] [Channel #2128 SubChannel #2129]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.112:50052", ServerName: "10.134.0.112:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.112:50052: operation was canceled" Nov 03 14:36:11.990692 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:11.986514 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-explicit-4a0349-5dd598dcf9-7fxc7" podUID="8804c6c5-0c17-455e-82b9-89dee267575e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.113:50052\" within 1s: context deadline exceeded" Nov 03 14:36:12.621002 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:12.620971 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3495e145-fcb4-4338-8965-1581c04a6e4b" path="/var/lib/kubelet/pods/3495e145-fcb4-4338-8965-1581c04a6e4b/volumes" Nov 03 14:36:12.621461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:12.621447 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8804c6c5-0c17-455e-82b9-89dee267575e" path="/var/lib/kubelet/pods/8804c6c5-0c17-455e-82b9-89dee267575e/volumes" Nov 03 14:36:21.558121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:21.558082 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd"] Nov 03 14:36:21.558548 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:21.558437 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" podUID="ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a" containerName="maas-api" containerID="cri-o://2868d259b9a2cfbccdc8c58fd8223eaa2155ca1f81fd6a0156238b2926dbaae9" gracePeriod=30 Nov 03 14:36:21.965400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:21.965024 6389 generic.go:356] "Generic (PLEG): container finished" podID="ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a" containerID="2868d259b9a2cfbccdc8c58fd8223eaa2155ca1f81fd6a0156238b2926dbaae9" exitCode=0 Nov 03 14:36:21.965400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:21.965190 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" event={"ID":"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a","Type":"ContainerDied","Data":"2868d259b9a2cfbccdc8c58fd8223eaa2155ca1f81fd6a0156238b2926dbaae9"} Nov 03 14:36:21.965400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:21.965263 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" event={"ID":"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a","Type":"ContainerDied","Data":"a8c7d3d587f55341dfaa6a0a5f39fad822b508df4126a1fb5f5cbd11463cb741"} Nov 03 14:36:21.965400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:21.965281 6389 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8c7d3d587f55341dfaa6a0a5f39fad822b508df4126a1fb5f5cbd11463cb741" Nov 03 14:36:21.968959 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:21.968675 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:36:22.069638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.062103 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-kube-api-access-bjhrr\" (UniqueName: \"kubernetes.io/projected/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-kube-api-access-bjhrr\") pod \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " Nov 03 14:36:22.069638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.062328 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-maas-api-tls\") pod \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " Nov 03 14:36:22.069638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.062385 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-openshift-service-ca\") pod \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " Nov 03 14:36:22.069638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.062447 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-metrics-certs\") pod \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\" (UID: \"ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a\") " Nov 03 14:36:22.069638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.065210 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-openshift-service-ca" pod "ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a" (UID: "ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:36:22.099665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.097288 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-kube-api-access-bjhrr" pod "ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a" (UID: "ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a"). InnerVolumeSpecName "kube-api-access-bjhrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:22.099841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.099707 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-metrics-certs" pod "ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a" (UID: "ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:36:22.099841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.099737 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-maas-api-tls" pod "ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a" (UID: "ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:36:22.196879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.196837 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjhrr\" (UniqueName: \"kubernetes.io/projected/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-kube-api-access-bjhrr\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:22.197188 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.197162 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:22.197345 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.197328 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:22.197452 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.197438 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:22.992650 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:22.991254 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd" Nov 03 14:36:23.084802 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:23.084764 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd"] Nov 03 14:36:23.107645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:23.095527 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4a0349-7d8978658d-c92nd"] Nov 03 14:36:24.623099 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:24.623060 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a" path="/var/lib/kubelet/pods/ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a/volumes" Nov 03 14:36:27.013769 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.013725 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms"] Nov 03 14:36:27.014592 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014570 6389 cpu_manager.go:414] "containerMap: removing container" podUID="8804c6c5-0c17-455e-82b9-89dee267575e" containerName="payload-processing" Nov 03 14:36:27.014592 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014591 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8804c6c5-0c17-455e-82b9-89dee267575e" containerName="payload-processing" Nov 03 14:36:27.014800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014640 6389 cpu_manager.go:414] "containerMap: removing container" podUID="a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1" containerName="payload-processing" Nov 03 14:36:27.014800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014647 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1" containerName="payload-processing" Nov 03 14:36:27.014800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014657 6389 cpu_manager.go:414] "containerMap: removing container" podUID="3495e145-fcb4-4338-8965-1581c04a6e4b" containerName="payload-pre-processing" Nov 03 14:36:27.014800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014663 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3495e145-fcb4-4338-8965-1581c04a6e4b" containerName="payload-pre-processing" Nov 03 14:36:27.014800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014674 6389 cpu_manager.go:414] "containerMap: removing container" podUID="c31f97c3-0499-4900-bd83-9c0cfe4a3c5c" containerName="payload-processing" Nov 03 14:36:27.014800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014678 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31f97c3-0499-4900-bd83-9c0cfe4a3c5c" containerName="payload-processing" Nov 03 14:36:27.014800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014692 6389 cpu_manager.go:414] "containerMap: removing container" podUID="55e3b5f4-8a45-4c0e-a370-f37e9da34eda" containerName="payload-pre-processing" Nov 03 14:36:27.014800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014696 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e3b5f4-8a45-4c0e-a370-f37e9da34eda" containerName="payload-pre-processing" Nov 03 14:36:27.014800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014707 6389 cpu_manager.go:414] "containerMap: removing container" podUID="ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a" containerName="maas-api" Nov 03 14:36:27.014800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014711 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a" containerName="maas-api" Nov 03 14:36:27.014800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014806 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31f97c3-0499-4900-bd83-9c0cfe4a3c5c" containerName="payload-processing" Nov 03 14:36:27.015378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014816 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="8804c6c5-0c17-455e-82b9-89dee267575e" containerName="payload-processing" Nov 03 14:36:27.015378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014823 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e3b5f4-8a45-4c0e-a370-f37e9da34eda" containerName="payload-pre-processing" Nov 03 14:36:27.015378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014832 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed90a8d6-c348-4e7c-add4-c5bbb7d91e0a" containerName="maas-api" Nov 03 14:36:27.015378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014839 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4cf9c8e-dbff-4ebc-9885-feb6d4cc8ea1" containerName="payload-processing" Nov 03 14:36:27.015378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.014845 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="3495e145-fcb4-4338-8965-1581c04a6e4b" containerName="payload-pre-processing" Nov 03 14:36:27.024351 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.024324 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.047528 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.047496 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms"] Nov 03 14:36:27.048896 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.048069 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-682a07-openshift-default-dockercfg-zcp5k\"" Nov 03 14:36:27.252122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.252082 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9mdj\" (UniqueName: \"kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-kube-api-access-c9mdj\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.271883 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.271720 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-certs\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.288993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.288951 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-data\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.290742 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.290716 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-istio-token\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.290983 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.290965 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-credential-socket\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.485766 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.485720 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-envoy\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.502458 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.502428 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-socket\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.502752 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.502727 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094f929c-8243-427f-a6fc-e740a20474e1-istio-podinfo\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.502944 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.502926 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094f929c-8243-427f-a6fc-e740a20474e1-istiod-ca-cert\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.604293 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.604198 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mdj\" (UniqueName: \"kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-kube-api-access-c9mdj\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.604575 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.604554 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-certs\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.604834 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.604811 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-data\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.604969 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.604950 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-istio-token\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.605098 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.605083 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-credential-socket\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.605226 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.605211 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-envoy\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.605355 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.605339 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-socket\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.605472 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.605456 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094f929c-8243-427f-a6fc-e740a20474e1-istio-podinfo\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.605658 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.605602 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094f929c-8243-427f-a6fc-e740a20474e1-istiod-ca-cert\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.605900 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.605151 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-certs\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.606293 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.606132 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-credential-socket\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.606293 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.605464 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-data\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.607224 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.607195 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-socket\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.607889 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.607868 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094f929c-8243-427f-a6fc-e740a20474e1-istiod-ca-cert\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.609940 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.609878 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-envoy\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.610382 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.610357 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094f929c-8243-427f-a6fc-e740a20474e1-istio-podinfo\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.701636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.681475 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mdj\" (UniqueName: \"kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-kube-api-access-c9mdj\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.701636 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.696921 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-istio-token\") pod \"e2e-noten-682a07-openshift-default-85774c99b9-hrdms\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:27.991922 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:27.991867 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:28.538671 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:36:28.535413 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:36:28.599026 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:28.598997 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms"] Nov 03 14:36:28.604918 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:36:28.604501 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094f929c_8243_427f_a6fc_e740a20474e1.slice/crio-df7dea29c3a54955ee5fc1d290510c70df5681cdb3d37b1852218b2c70eb4192 WatchSource:0}: Error finding container df7dea29c3a54955ee5fc1d290510c70df5681cdb3d37b1852218b2c70eb4192: Status 404 returned error can't find the container with id df7dea29c3a54955ee5fc1d290510c70df5681cdb3d37b1852218b2c70eb4192 Nov 03 14:36:28.608512 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:28.608389 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:36:28.608512 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:28.608472 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:36:28.608512 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:28.608510 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:36:29.381285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:29.381245 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" event={"ID":"094f929c-8243-427f-a6fc-e740a20474e1","Type":"ContainerStarted","Data":"32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f"} Nov 03 14:36:29.381519 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:29.381309 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" event={"ID":"094f929c-8243-427f-a6fc-e740a20474e1","Type":"ContainerStarted","Data":"df7dea29c3a54955ee5fc1d290510c70df5681cdb3d37b1852218b2c70eb4192"} Nov 03 14:36:29.441174 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:29.441121 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" podStartSLOduration=3.441107731 podStartE2EDuration="3.441107731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:36:29.440401983 +0000 UTC m=+1464.999038466" watchObservedRunningTime="2026-11-03 14:36:29.441107731 +0000 UTC m=+1464.999744268" Nov 03 14:36:29.992866 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:29.992831 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:29.998161 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:29.998133 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:30.387529 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:30.387455 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:30.388353 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:30.388336 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:39.915813 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:39.915776 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m"] Nov 03 14:36:39.916328 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:39.916047 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" containerID="cri-o://74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d" gracePeriod=30 Nov 03 14:36:40.652116 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:40.652074 6389 patch_prober.go:29] interesting pod/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:36:40.652321 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:40.652144 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:36:45.204804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.204779 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:36:45.389308 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.389278 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-certs\") pod \"09311533-6310-4062-b1a2-1167c4e4bbc2\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " Nov 03 14:36:45.389526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.389352 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-kube-api-access-ndl5l\" (UniqueName: \"kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-kube-api-access-ndl5l\") pod \"09311533-6310-4062-b1a2-1167c4e4bbc2\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " Nov 03 14:36:45.389526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.389433 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-credential-socket\") pod \"09311533-6310-4062-b1a2-1167c4e4bbc2\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " Nov 03 14:36:45.389526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.389485 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09311533-6310-4062-b1a2-1167c4e4bbc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09311533-6310-4062-b1a2-1167c4e4bbc2-istiod-ca-cert\") pod \"09311533-6310-4062-b1a2-1167c4e4bbc2\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " Nov 03 14:36:45.389526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.389513 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-data\") pod \"09311533-6310-4062-b1a2-1167c4e4bbc2\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " Nov 03 14:36:45.389847 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.389551 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-socket\") pod \"09311533-6310-4062-b1a2-1167c4e4bbc2\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " Nov 03 14:36:45.389847 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.389642 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-certs" pod "09311533-6310-4062-b1a2-1167c4e4bbc2" (UID: "09311533-6310-4062-b1a2-1167c4e4bbc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:45.389847 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.389657 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-podinfo\") pod \"09311533-6310-4062-b1a2-1167c4e4bbc2\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " Nov 03 14:36:45.389847 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.389721 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-token\" (UniqueName: \"kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-token\") pod \"09311533-6310-4062-b1a2-1167c4e4bbc2\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " Nov 03 14:36:45.389847 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.389762 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-envoy\") pod \"09311533-6310-4062-b1a2-1167c4e4bbc2\" (UID: \"09311533-6310-4062-b1a2-1167c4e4bbc2\") " Nov 03 14:36:45.389847 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.389835 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-credential-socket" pod "09311533-6310-4062-b1a2-1167c4e4bbc2" (UID: "09311533-6310-4062-b1a2-1167c4e4bbc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:45.390123 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.389873 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-socket" pod "09311533-6310-4062-b1a2-1167c4e4bbc2" (UID: "09311533-6310-4062-b1a2-1167c4e4bbc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:45.390123 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.390019 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-data" pod "09311533-6310-4062-b1a2-1167c4e4bbc2" (UID: "09311533-6310-4062-b1a2-1167c4e4bbc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:45.390229 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.390167 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09311533-6310-4062-b1a2-1167c4e4bbc2-istiod-ca-cert" pod "09311533-6310-4062-b1a2-1167c4e4bbc2" (UID: "09311533-6310-4062-b1a2-1167c4e4bbc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:36:45.390484 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.390461 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:45.390484 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.390484 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:45.390720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.390498 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09311533-6310-4062-b1a2-1167c4e4bbc2-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:45.390720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.390511 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:45.390720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.390550 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:45.392524 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.392500 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-podinfo" pod "09311533-6310-4062-b1a2-1167c4e4bbc2" (UID: "09311533-6310-4062-b1a2-1167c4e4bbc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:36:45.392653 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.392635 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-token" pod "09311533-6310-4062-b1a2-1167c4e4bbc2" (UID: "09311533-6310-4062-b1a2-1167c4e4bbc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:45.392727 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.392680 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-envoy" pod "09311533-6310-4062-b1a2-1167c4e4bbc2" (UID: "09311533-6310-4062-b1a2-1167c4e4bbc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:45.392727 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.392698 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-kube-api-access-ndl5l" pod "09311533-6310-4062-b1a2-1167c4e4bbc2" (UID: "09311533-6310-4062-b1a2-1167c4e4bbc2"). InnerVolumeSpecName "kube-api-access-ndl5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:45.492218 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.492140 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:45.492218 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.492168 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:45.492218 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.492177 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09311533-6310-4062-b1a2-1167c4e4bbc2-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:45.492218 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.492186 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndl5l\" (UniqueName: \"kubernetes.io/projected/09311533-6310-4062-b1a2-1167c4e4bbc2-kube-api-access-ndl5l\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:45.513892 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.513856 6389 generic.go:356] "Generic (PLEG): container finished" podID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerID="74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d" exitCode=0 Nov 03 14:36:45.514039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.513913 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" event={"ID":"09311533-6310-4062-b1a2-1167c4e4bbc2","Type":"ContainerDied","Data":"74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d"} Nov 03 14:36:45.514039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.513914 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" Nov 03 14:36:45.514039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.513994 6389 scope.go:122] "RemoveContainer" containerID="74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d" Nov 03 14:36:45.514147 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.513984 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m" event={"ID":"09311533-6310-4062-b1a2-1167c4e4bbc2","Type":"ContainerDied","Data":"ef94f2f1732bafd931e0997893c58c68fa509115a644399f90b85d0347747678"} Nov 03 14:36:45.527344 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.527260 6389 scope.go:122] "RemoveContainer" containerID="21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9" Nov 03 14:36:45.536922 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.536904 6389 scope.go:122] "RemoveContainer" containerID="74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d" Nov 03 14:36:45.537179 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:36:45.537161 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d\": container with ID starting with 74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d not found: ID does not exist" containerID="74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d" Nov 03 14:36:45.537229 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.537189 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d"} err="failed to get container status \"74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d\": rpc error: code = NotFound desc = could not find container \"74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d\": container with ID starting with 74d0e4eebc4f4219e505a14642e0c9af2ca383d964718df1a703402b34e8a25d not found: ID does not exist" Nov 03 14:36:45.537229 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.537204 6389 scope.go:122] "RemoveContainer" containerID="21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9" Nov 03 14:36:45.537452 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:36:45.537434 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9\": container with ID starting with 21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9 not found: ID does not exist" containerID="21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9" Nov 03 14:36:45.537497 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.537458 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9"} err="failed to get container status \"21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9\": rpc error: code = NotFound desc = could not find container \"21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9\": container with ID starting with 21dfdc32b74b17dadd5e552bb4109d1048909b2dd5b529b03c78ad31a16242d9 not found: ID does not exist" Nov 03 14:36:45.577391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.577362 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m"] Nov 03 14:36:45.584164 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:45.584141 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2e8614-openshift-default-6784dcddf9-pwf5m"] Nov 03 14:36:46.620892 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:46.620856 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" path="/var/lib/kubelet/pods/09311533-6310-4062-b1a2-1167c4e4bbc2/volumes" Nov 03 14:36:47.330169 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:47.330138 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms"] Nov 03 14:36:47.330425 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:47.330398 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" podUID="094f929c-8243-427f-a6fc-e740a20474e1" containerName="istio-proxy" containerID="cri-o://32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f" gracePeriod=30 Nov 03 14:36:52.597529 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.597472 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:52.631344 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.631316 6389 generic.go:356] "Generic (PLEG): container finished" podID="094f929c-8243-427f-a6fc-e740a20474e1" containerID="32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f" exitCode=0 Nov 03 14:36:52.631508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.631369 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" Nov 03 14:36:52.631508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.631382 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" event={"ID":"094f929c-8243-427f-a6fc-e740a20474e1","Type":"ContainerDied","Data":"32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f"} Nov 03 14:36:52.631508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.631420 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms" event={"ID":"094f929c-8243-427f-a6fc-e740a20474e1","Type":"ContainerDied","Data":"df7dea29c3a54955ee5fc1d290510c70df5681cdb3d37b1852218b2c70eb4192"} Nov 03 14:36:52.631508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.631442 6389 scope.go:122] "RemoveContainer" containerID="32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f" Nov 03 14:36:52.634912 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.634887 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-envoy\") pod \"094f929c-8243-427f-a6fc-e740a20474e1\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " Nov 03 14:36:52.635018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.634940 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-credential-socket\") pod \"094f929c-8243-427f-a6fc-e740a20474e1\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " Nov 03 14:36:52.635018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.634981 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-istio-token\" (UniqueName: \"kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-istio-token\") pod \"094f929c-8243-427f-a6fc-e740a20474e1\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " Nov 03 14:36:52.635123 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.635056 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-certs\") pod \"094f929c-8243-427f-a6fc-e740a20474e1\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " Nov 03 14:36:52.635123 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.635107 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-data\") pod \"094f929c-8243-427f-a6fc-e740a20474e1\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " Nov 03 14:36:52.635236 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.635147 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/094f929c-8243-427f-a6fc-e740a20474e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094f929c-8243-427f-a6fc-e740a20474e1-istio-podinfo\") pod \"094f929c-8243-427f-a6fc-e740a20474e1\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " Nov 03 14:36:52.635296 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.635232 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-socket\") pod \"094f929c-8243-427f-a6fc-e740a20474e1\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " Nov 03 14:36:52.635296 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.635266 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-credential-socket" pod "094f929c-8243-427f-a6fc-e740a20474e1" (UID: "094f929c-8243-427f-a6fc-e740a20474e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:52.635417 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.635319 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-kube-api-access-c9mdj\" (UniqueName: \"kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-kube-api-access-c9mdj\") pod \"094f929c-8243-427f-a6fc-e740a20474e1\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " Nov 03 14:36:52.635486 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.635459 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094f929c-8243-427f-a6fc-e740a20474e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094f929c-8243-427f-a6fc-e740a20474e1-istiod-ca-cert\") pod \"094f929c-8243-427f-a6fc-e740a20474e1\" (UID: \"094f929c-8243-427f-a6fc-e740a20474e1\") " Nov 03 14:36:52.636514 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.635432 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-certs" pod "094f929c-8243-427f-a6fc-e740a20474e1" (UID: "094f929c-8243-427f-a6fc-e740a20474e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:52.636514 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.635486 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-data" pod "094f929c-8243-427f-a6fc-e740a20474e1" (UID: "094f929c-8243-427f-a6fc-e740a20474e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:52.636514 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.635641 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-socket" pod "094f929c-8243-427f-a6fc-e740a20474e1" (UID: "094f929c-8243-427f-a6fc-e740a20474e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:52.636514 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.635997 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:52.636514 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.636016 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:52.636514 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.636029 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:52.636514 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.636042 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:52.637326 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.637304 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094f929c-8243-427f-a6fc-e740a20474e1-istiod-ca-cert" pod "094f929c-8243-427f-a6fc-e740a20474e1" (UID: "094f929c-8243-427f-a6fc-e740a20474e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:36:52.637890 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.637871 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-istio-token" pod "094f929c-8243-427f-a6fc-e740a20474e1" (UID: "094f929c-8243-427f-a6fc-e740a20474e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:52.638469 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.638447 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-envoy" pod "094f929c-8243-427f-a6fc-e740a20474e1" (UID: "094f929c-8243-427f-a6fc-e740a20474e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:52.638550 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.638469 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-kube-api-access-c9mdj" pod "094f929c-8243-427f-a6fc-e740a20474e1" (UID: "094f929c-8243-427f-a6fc-e740a20474e1"). InnerVolumeSpecName "kube-api-access-c9mdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:52.638550 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.638530 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/094f929c-8243-427f-a6fc-e740a20474e1-istio-podinfo" pod "094f929c-8243-427f-a6fc-e740a20474e1" (UID: "094f929c-8243-427f-a6fc-e740a20474e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:36:52.644667 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.644647 6389 scope.go:122] "RemoveContainer" containerID="32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f" Nov 03 14:36:52.644949 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:36:52.644932 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f\": container with ID starting with 32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f not found: ID does not exist" containerID="32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f" Nov 03 14:36:52.645018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.644960 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f"} err="failed to get container status \"32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f\": rpc error: code = NotFound desc = could not find container \"32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f\": container with ID starting with 32a51e4f64704208474e45f549cb2dfd90e5db4f5c690e97e10121e12793d45f not found: ID does not exist" Nov 03 14:36:52.743634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.742771 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094f929c-8243-427f-a6fc-e740a20474e1-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:52.743634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.742812 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094f929c-8243-427f-a6fc-e740a20474e1-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:52.743634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.742830 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:52.743634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.742847 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094f929c-8243-427f-a6fc-e740a20474e1-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:52.743634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.742911 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9mdj\" (UniqueName: \"kubernetes.io/projected/094f929c-8243-427f-a6fc-e740a20474e1-kube-api-access-c9mdj\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:36:52.982824 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.982789 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms"] Nov 03 14:36:52.997051 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:52.997016 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-682a07-openshift-default-85774c99b9-hrdms"] Nov 03 14:36:54.621579 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:54.621542 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094f929c-8243-427f-a6fc-e740a20474e1" path="/var/lib/kubelet/pods/094f929c-8243-427f-a6fc-e740a20474e1/volumes" Nov 03 14:36:58.413624 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:36:58.413576 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:36:59.815774 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:59.815729 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:36:59.816220 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:59.815805 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:36:59.954635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:59.954119 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:36:59.954635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:36:59.954217 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:37:28.375215 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:37:28.375170 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:37:33.388622 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.388573 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw"] Nov 03 14:37:33.389163 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.389149 6389 cpu_manager.go:414] "containerMap: removing container" podUID="094f929c-8243-427f-a6fc-e740a20474e1" containerName="istio-proxy" Nov 03 14:37:33.389212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.389164 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094f929c-8243-427f-a6fc-e740a20474e1" containerName="istio-proxy" Nov 03 14:37:33.389212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.389179 6389 cpu_manager.go:414] "containerMap: removing container" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" Nov 03 14:37:33.389212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.389183 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" Nov 03 14:37:33.389212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.389202 6389 cpu_manager.go:414] "containerMap: removing container" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" Nov 03 14:37:33.389212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.389206 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" Nov 03 14:37:33.389358 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.389303 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="094f929c-8243-427f-a6fc-e740a20474e1" containerName="istio-proxy" Nov 03 14:37:33.389358 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.389311 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" Nov 03 14:37:33.392935 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.392915 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.443566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.443533 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 14:37:33.443840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.443734 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7469958f157e\"" Nov 03 14:37:33.443840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.443748 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 14:37:33.444004 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.443749 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-c4fsz\"" Nov 03 14:37:33.444004 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.443828 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 14:37:33.444004 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.443888 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 14:37:33.448600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.448573 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw"] Nov 03 14:37:33.514323 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.514278 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-routing-overlay-volume\") pod \"payload-processing-external-model-689fc7c65b-4wrkw\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.514491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.514352 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-plugins-config-volume\") pod \"payload-processing-external-model-689fc7c65b-4wrkw\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.514491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.514460 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tntv\" (UniqueName: \"kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-kube-api-access-4tntv\") pod \"payload-processing-external-model-689fc7c65b-4wrkw\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.514720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.514696 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-provider-credentials\") pod \"payload-processing-external-model-689fc7c65b-4wrkw\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.616572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.616541 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-provider-credentials\") pod \"payload-processing-external-model-689fc7c65b-4wrkw\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.616748 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.616645 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-routing-overlay-volume\") pod \"payload-processing-external-model-689fc7c65b-4wrkw\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.616748 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.616679 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-plugins-config-volume\") pod \"payload-processing-external-model-689fc7c65b-4wrkw\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.616748 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.616710 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tntv\" (UniqueName: \"kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-kube-api-access-4tntv\") pod \"payload-processing-external-model-689fc7c65b-4wrkw\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.617461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.617429 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-routing-overlay-volume\") pod \"payload-processing-external-model-689fc7c65b-4wrkw\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.617591 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.617441 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-plugins-config-volume\") pod \"payload-processing-external-model-689fc7c65b-4wrkw\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.619251 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.619229 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-provider-credentials\") pod \"payload-processing-external-model-689fc7c65b-4wrkw\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.641208 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.641138 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tntv\" (UniqueName: \"kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-kube-api-access-4tntv\") pod \"payload-processing-external-model-689fc7c65b-4wrkw\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:33.709176 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:33.708856 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:34.065770 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:34.065737 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw"] Nov 03 14:37:34.067164 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:37:34.067139 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc039791a_d490_415d_9c6f_467a6ca9552a.slice/crio-e3295b2ab6c52e5d6455e7edb4ce4beb50f96ff88099089f915ac96d89066110 WatchSource:0}: Error finding container e3295b2ab6c52e5d6455e7edb4ce4beb50f96ff88099089f915ac96d89066110: Status 404 returned error can't find the container with id e3295b2ab6c52e5d6455e7edb4ce4beb50f96ff88099089f915ac96d89066110 Nov 03 14:37:34.105741 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:34.105713 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" event={"ID":"c039791a-d490-415d-9c6f-467a6ca9552a","Type":"ContainerStarted","Data":"e3295b2ab6c52e5d6455e7edb4ce4beb50f96ff88099089f915ac96d89066110"} Nov 03 14:37:35.112921 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:35.112883 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" event={"ID":"c039791a-d490-415d-9c6f-467a6ca9552a","Type":"ContainerStarted","Data":"cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7"} Nov 03 14:37:35.113381 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:35.112951 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:35.158021 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:35.157971 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" podStartSLOduration=2.157958564 podStartE2EDuration="2.157958564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:35.155454388 +0000 UTC m=+1530.714090874" watchObservedRunningTime="2026-11-03 14:37:35.157958564 +0000 UTC m=+1530.716595048" Nov 03 14:37:37.208001 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:37.207955 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f"] Nov 03 14:37:37.237966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:37.237932 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59"] Nov 03 14:37:45.462546 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:45.462449 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw"] Nov 03 14:37:45.465948 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:45.465925 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:50.463808 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:50.463732 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" podUID="c039791a-d490-415d-9c6f-467a6ca9552a" containerName="payload-processing" containerID="cri-o://cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7" gracePeriod=25 Nov 03 14:37:50.746379 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:50.746354 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:50.847635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:50.833132 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-routing-overlay-volume\") pod \"c039791a-d490-415d-9c6f-467a6ca9552a\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " Nov 03 14:37:50.874635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:50.873748 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-kube-api-access-4tntv\" (UniqueName: \"kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-kube-api-access-4tntv\") pod \"c039791a-d490-415d-9c6f-467a6ca9552a\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " Nov 03 14:37:50.874635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:50.873956 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-provider-credentials\") pod \"c039791a-d490-415d-9c6f-467a6ca9552a\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " Nov 03 14:37:50.880720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:50.861840 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-routing-overlay-volume" pod "c039791a-d490-415d-9c6f-467a6ca9552a" (UID: "c039791a-d490-415d-9c6f-467a6ca9552a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:37:50.886726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:50.886688 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-plugins-config-volume\") pod \"c039791a-d490-415d-9c6f-467a6ca9552a\" (UID: \"c039791a-d490-415d-9c6f-467a6ca9552a\") " Nov 03 14:37:50.901643 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:50.899362 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:50.901643 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:50.899938 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-plugins-config-volume" pod "c039791a-d490-415d-9c6f-467a6ca9552a" (UID: "c039791a-d490-415d-9c6f-467a6ca9552a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:37:50.979719 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:50.974344 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-kube-api-access-4tntv" pod "c039791a-d490-415d-9c6f-467a6ca9552a" (UID: "c039791a-d490-415d-9c6f-467a6ca9552a"). InnerVolumeSpecName "kube-api-access-4tntv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:37:51.002773 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.002726 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tntv\" (UniqueName: \"kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-kube-api-access-4tntv\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:51.002773 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.002768 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c039791a-d490-415d-9c6f-467a6ca9552a-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:51.058956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.058914 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-provider-credentials" pod "c039791a-d490-415d-9c6f-467a6ca9552a" (UID: "c039791a-d490-415d-9c6f-467a6ca9552a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:37:51.125635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.109719 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c039791a-d490-415d-9c6f-467a6ca9552a-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:51.279690 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.279416 6389 generic.go:356] "Generic (PLEG): container finished" podID="c039791a-d490-415d-9c6f-467a6ca9552a" containerID="cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7" exitCode=0 Nov 03 14:37:51.290634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.279915 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" event={"ID":"c039791a-d490-415d-9c6f-467a6ca9552a","Type":"ContainerDied","Data":"cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7"} Nov 03 14:37:51.290634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.279981 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" event={"ID":"c039791a-d490-415d-9c6f-467a6ca9552a","Type":"ContainerDied","Data":"e3295b2ab6c52e5d6455e7edb4ce4beb50f96ff88099089f915ac96d89066110"} Nov 03 14:37:51.290634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.280014 6389 scope.go:122] "RemoveContainer" containerID="cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7" Nov 03 14:37:51.290634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.280176 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw" Nov 03 14:37:51.438640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.419208 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw"] Nov 03 14:37:51.455635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.447018 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-689fc7c65b-4wrkw"] Nov 03 14:37:51.473693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.473020 6389 scope.go:122] "RemoveContainer" containerID="cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7" Nov 03 14:37:51.473693 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:37:51.473555 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7\": container with ID starting with cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7 not found: ID does not exist" containerID="cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7" Nov 03 14:37:51.473693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:51.473594 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7"} err="failed to get container status \"cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7\": rpc error: code = NotFound desc = could not find container \"cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7\": container with ID starting with cfcdf7f17e18582b96fa885bab065bcde64eb532e6c1e675d2a712613d8335a7 not found: ID does not exist" Nov 03 14:37:52.304637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:52.294766 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" podUID="76e983d7-ca69-44d1-857c-65eb03ec5e6a" containerName="main" containerID="cri-o://78208333ea53d9195e7016e168d7d3f10353db9d028157238d9c725f50c17519" gracePeriod=45 Nov 03 14:37:52.304637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:52.297760 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" podUID="151733fd-8ae3-4401-870c-8384e18d768b" containerName="main" containerID="cri-o://4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc" gracePeriod=45 Nov 03 14:37:52.377821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:52.377707 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" podUID="151733fd-8ae3-4401-870c-8384e18d768b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": EOF" Nov 03 14:37:52.479149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:52.472823 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" podUID="76e983d7-ca69-44d1-857c-65eb03ec5e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": EOF" Nov 03 14:37:52.690967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:52.690084 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c039791a-d490-415d-9c6f-467a6ca9552a" path="/var/lib/kubelet/pods/c039791a-d490-415d-9c6f-467a6ca9552a/volumes" Nov 03 14:37:53.259068 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:53.259027 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" podUID="76e983d7-ca69-44d1-857c-65eb03ec5e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": dial tcp 10.134.0.47:8000: connect: connection refused" Nov 03 14:37:53.259441 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:53.259419 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:37:53.540694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:53.308323 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88"] Nov 03 14:37:53.782949 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:53.624753 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw"] Nov 03 14:37:54.119328 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.119291 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:37:54.151655 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.132680 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-tmp-dir\") pod \"151733fd-8ae3-4401-870c-8384e18d768b\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " Nov 03 14:37:54.151655 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.132733 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-home\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-home\") pod \"151733fd-8ae3-4401-870c-8384e18d768b\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " Nov 03 14:37:54.151655 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.132777 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/151733fd-8ae3-4401-870c-8384e18d768b-tls-certs\" (UniqueName: \"kubernetes.io/secret/151733fd-8ae3-4401-870c-8384e18d768b-tls-certs\") pod \"151733fd-8ae3-4401-870c-8384e18d768b\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " Nov 03 14:37:54.151655 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.132922 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-dshm\") pod \"151733fd-8ae3-4401-870c-8384e18d768b\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " Nov 03 14:37:54.151655 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.133032 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151733fd-8ae3-4401-870c-8384e18d768b-kube-api-access-68jzw\" (UniqueName: \"kubernetes.io/projected/151733fd-8ae3-4401-870c-8384e18d768b-kube-api-access-68jzw\") pod \"151733fd-8ae3-4401-870c-8384e18d768b\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " Nov 03 14:37:54.151655 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.133086 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-model-cache\") pod \"151733fd-8ae3-4401-870c-8384e18d768b\" (UID: \"151733fd-8ae3-4401-870c-8384e18d768b\") " Nov 03 14:37:54.176450 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.176411 6389 generic.go:356] "Generic (PLEG): container finished" podID="76e983d7-ca69-44d1-857c-65eb03ec5e6a" containerID="78208333ea53d9195e7016e168d7d3f10353db9d028157238d9c725f50c17519" exitCode=0 Nov 03 14:37:54.177056 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.177027 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-model-cache" pod "151733fd-8ae3-4401-870c-8384e18d768b" (UID: "151733fd-8ae3-4401-870c-8384e18d768b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:54.179696 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.177203 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" event={"ID":"76e983d7-ca69-44d1-857c-65eb03ec5e6a","Type":"ContainerDied","Data":"78208333ea53d9195e7016e168d7d3f10353db9d028157238d9c725f50c17519"} Nov 03 14:37:54.179696 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.178896 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-tmp-dir" pod "151733fd-8ae3-4401-870c-8384e18d768b" (UID: "151733fd-8ae3-4401-870c-8384e18d768b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:54.179696 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.179259 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-home" pod "151733fd-8ae3-4401-870c-8384e18d768b" (UID: "151733fd-8ae3-4401-870c-8384e18d768b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:54.296633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.291684 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" podUID="76e983d7-ca69-44d1-857c-65eb03ec5e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": dial tcp 10.134.0.47:8000: connect: connection refused" Nov 03 14:37:54.296633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.291700 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" podUID="151733fd-8ae3-4401-870c-8384e18d768b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:37:54.296633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.291876 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:37:54.320813 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.320660 6389 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-model-cache\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:54.320813 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.320714 6389 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-tmp-dir\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:54.320813 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.320727 6389 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-home\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:54.324283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.323875 6389 generic.go:356] "Generic (PLEG): container finished" podID="151733fd-8ae3-4401-870c-8384e18d768b" containerID="4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc" exitCode=0 Nov 03 14:37:54.324283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.323962 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" event={"ID":"151733fd-8ae3-4401-870c-8384e18d768b","Type":"ContainerDied","Data":"4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc"} Nov 03 14:37:54.324283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.323998 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" event={"ID":"151733fd-8ae3-4401-870c-8384e18d768b","Type":"ContainerDied","Data":"312fc0fc64126f340e0a75973a355096230a391d018f09d70f13e91eb5cb9eba"} Nov 03 14:37:54.324283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.324021 6389 scope.go:122] "RemoveContainer" containerID="4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc" Nov 03 14:37:54.324283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.324141 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f" Nov 03 14:37:54.376836 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.376729 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-dshm" pod "151733fd-8ae3-4401-870c-8384e18d768b" (UID: "151733fd-8ae3-4401-870c-8384e18d768b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:54.442652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.422788 6389 scope.go:122] "RemoveContainer" containerID="4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc" Nov 03 14:37:54.442652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.426703 6389 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151733fd-8ae3-4401-870c-8384e18d768b-dshm\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:54.444080 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:37:54.443772 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc\": container with ID starting with 4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc not found: ID does not exist" containerID="4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc" Nov 03 14:37:54.444080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.443819 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc"} err="failed to get container status \"4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc\": rpc error: code = NotFound desc = could not find container \"4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc\": container with ID starting with 4caec80f861ad9d86fe9e8b471f39565837d756b52a53042bc1fe2bcb41a7acc not found: ID does not exist" Nov 03 14:37:54.480839 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.480762 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151733fd-8ae3-4401-870c-8384e18d768b-tls-certs" pod "151733fd-8ae3-4401-870c-8384e18d768b" (UID: "151733fd-8ae3-4401-870c-8384e18d768b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:37:54.521432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.521369 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151733fd-8ae3-4401-870c-8384e18d768b-kube-api-access-68jzw" pod "151733fd-8ae3-4401-870c-8384e18d768b" (UID: "151733fd-8ae3-4401-870c-8384e18d768b"). InnerVolumeSpecName "kube-api-access-68jzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:37:54.659726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.599848 6389 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151733fd-8ae3-4401-870c-8384e18d768b-tls-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:54.659726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:54.599892 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68jzw\" (UniqueName: \"kubernetes.io/projected/151733fd-8ae3-4401-870c-8384e18d768b-kube-api-access-68jzw\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:55.450515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.450484 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f"] Nov 03 14:37:55.453748 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.453713 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/distinct-2-w0-86c2e2-kserve-cf4cf7d6b-7ln8f"] Nov 03 14:37:55.466824 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:37:55.466783 6389 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod151733fd_8ae3_4401_870c_8384e18d768b.slice/crio-312fc0fc64126f340e0a75973a355096230a391d018f09d70f13e91eb5cb9eba\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod151733fd_8ae3_4401_870c_8384e18d768b.slice\": RecentStats: unable to find data in memory cache]" Nov 03 14:37:55.467924 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:37:55.466868 6389 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod151733fd_8ae3_4401_870c_8384e18d768b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod151733fd_8ae3_4401_870c_8384e18d768b.slice/crio-312fc0fc64126f340e0a75973a355096230a391d018f09d70f13e91eb5cb9eba\": RecentStats: unable to find data in memory cache]" Nov 03 14:37:55.476695 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.476660 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" event={"ID":"76e983d7-ca69-44d1-857c-65eb03ec5e6a","Type":"ContainerDied","Data":"0a29670979c565aca271e68b89fb71503107b2d212edbf0e1f0eaa6bb056804e"} Nov 03 14:37:55.476974 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.476955 6389 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a29670979c565aca271e68b89fb71503107b2d212edbf0e1f0eaa6bb056804e" Nov 03 14:37:55.484337 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.484271 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:37:55.651684 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.649495 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e983d7-ca69-44d1-857c-65eb03ec5e6a-kube-api-access-w7xvt\" (UniqueName: \"kubernetes.io/projected/76e983d7-ca69-44d1-857c-65eb03ec5e6a-kube-api-access-w7xvt\") pod \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " Nov 03 14:37:55.651684 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.649660 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-dshm\") pod \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " Nov 03 14:37:55.651684 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.649717 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tmp-dir\") pod \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " Nov 03 14:37:55.651684 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.649880 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-model-cache\") pod \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " Nov 03 14:37:55.651684 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.649969 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-home\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-home\") pod \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " Nov 03 14:37:55.651684 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.650075 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tls-certs\") pod \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\" (UID: \"76e983d7-ca69-44d1-857c-65eb03ec5e6a\") " Nov 03 14:37:55.652138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.652011 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tmp-dir" pod "76e983d7-ca69-44d1-857c-65eb03ec5e6a" (UID: "76e983d7-ca69-44d1-857c-65eb03ec5e6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:55.660683 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.658874 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-home" pod "76e983d7-ca69-44d1-857c-65eb03ec5e6a" (UID: "76e983d7-ca69-44d1-857c-65eb03ec5e6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:55.662187 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.662160 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-model-cache" pod "76e983d7-ca69-44d1-857c-65eb03ec5e6a" (UID: "76e983d7-ca69-44d1-857c-65eb03ec5e6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:55.682648 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.682038 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-dshm" pod "76e983d7-ca69-44d1-857c-65eb03ec5e6a" (UID: "76e983d7-ca69-44d1-857c-65eb03ec5e6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:55.683066 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.683040 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tls-certs" pod "76e983d7-ca69-44d1-857c-65eb03ec5e6a" (UID: "76e983d7-ca69-44d1-857c-65eb03ec5e6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:37:55.683436 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.683398 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e983d7-ca69-44d1-857c-65eb03ec5e6a-kube-api-access-w7xvt" pod "76e983d7-ca69-44d1-857c-65eb03ec5e6a" (UID: "76e983d7-ca69-44d1-857c-65eb03ec5e6a"). InnerVolumeSpecName "kube-api-access-w7xvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:37:55.752042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.752010 6389 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-dshm\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:55.752042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.752040 6389 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tmp-dir\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:55.752042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.752053 6389 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-model-cache\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:55.752288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.752064 6389 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e983d7-ca69-44d1-857c-65eb03ec5e6a-home\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:55.752288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.752075 6389 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e983d7-ca69-44d1-857c-65eb03ec5e6a-tls-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:55.752288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:55.752085 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7xvt\" (UniqueName: \"kubernetes.io/projected/76e983d7-ca69-44d1-857c-65eb03ec5e6a-kube-api-access-w7xvt\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:37:56.239336 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:56.239294 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" podUID="76e983d7-ca69-44d1-857c-65eb03ec5e6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 14:37:56.489572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:56.489495 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59" Nov 03 14:37:56.546647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:56.546600 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59"] Nov 03 14:37:56.559288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:56.559258 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/simulator-w0-86c2e2-kserve-fdf66f5f9-zrj59"] Nov 03 14:37:56.619251 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:56.619217 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151733fd-8ae3-4401-870c-8384e18d768b" path="/var/lib/kubelet/pods/151733fd-8ae3-4401-870c-8384e18d768b/volumes" Nov 03 14:37:56.619756 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:56.619740 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e983d7-ca69-44d1-857c-65eb03ec5e6a" path="/var/lib/kubelet/pods/76e983d7-ca69-44d1-857c-65eb03ec5e6a/volumes" Nov 03 14:37:58.374633 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:37:58.374575 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:37:58.625191 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:58.625105 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" podUID="2ce7aee1-fade-4b22-bf65-c3e6e6119db8" containerName="payload-processing" containerID="cri-o://152948f06ef3372801f448168e4cc34678ff250bbe2123cfc268fca4ab80e0dc" gracePeriod=25 Nov 03 14:37:58.625358 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:37:58.625201 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" podUID="0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6" containerName="payload-pre-processing" containerID="cri-o://823359b8ac65b6ff416da9649eaa08ed39c036abaa263d6d8ad7d8281e9b5b05" gracePeriod=25 Nov 03 14:38:02.337526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:02.337492 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" podUID="0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:38:03.342252 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:03.342215 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" podUID="2ce7aee1-fade-4b22-bf65-c3e6e6119db8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:38:04.673423 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:04.673391 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc"] Nov 03 14:38:04.674493 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:04.674440 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" podUID="2d0ce741-6676-438e-933b-47a0a19ee593" containerName="maas-api" containerID="cri-o://9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95" gracePeriod=30 Nov 03 14:38:05.133807 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.133778 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:38:05.223092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.223047 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0ce741-6676-438e-933b-47a0a19ee593-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ce741-6676-438e-933b-47a0a19ee593-openshift-service-ca\") pod \"2d0ce741-6676-438e-933b-47a0a19ee593\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " Nov 03 14:38:05.223782 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.223754 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0ce741-6676-438e-933b-47a0a19ee593-kube-api-access-9qzxh\" (UniqueName: \"kubernetes.io/projected/2d0ce741-6676-438e-933b-47a0a19ee593-kube-api-access-9qzxh\") pod \"2d0ce741-6676-438e-933b-47a0a19ee593\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " Nov 03 14:38:05.224592 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.224571 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-metrics-certs\") pod \"2d0ce741-6676-438e-933b-47a0a19ee593\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " Nov 03 14:38:05.234270 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.228555 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-metrics-certs" pod "2d0ce741-6676-438e-933b-47a0a19ee593" (UID: "2d0ce741-6676-438e-933b-47a0a19ee593"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:38:05.234714 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.234665 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-maas-api-tls" pod "2d0ce741-6676-438e-933b-47a0a19ee593" (UID: "2d0ce741-6676-438e-933b-47a0a19ee593"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:38:05.234936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.234920 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-maas-api-tls\") pod \"2d0ce741-6676-438e-933b-47a0a19ee593\" (UID: \"2d0ce741-6676-438e-933b-47a0a19ee593\") " Nov 03 14:38:05.249632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.235831 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:05.250117 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.250099 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0ce741-6676-438e-933b-47a0a19ee593-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:05.250293 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.250021 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0ce741-6676-438e-933b-47a0a19ee593-openshift-service-ca" pod "2d0ce741-6676-438e-933b-47a0a19ee593" (UID: "2d0ce741-6676-438e-933b-47a0a19ee593"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:05.266643 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.257863 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0ce741-6676-438e-933b-47a0a19ee593-kube-api-access-9qzxh" pod "2d0ce741-6676-438e-933b-47a0a19ee593" (UID: "2d0ce741-6676-438e-933b-47a0a19ee593"). InnerVolumeSpecName "kube-api-access-9qzxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:05.361470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.361334 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qzxh\" (UniqueName: \"kubernetes.io/projected/2d0ce741-6676-438e-933b-47a0a19ee593-kube-api-access-9qzxh\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:05.361470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.361371 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ce741-6676-438e-933b-47a0a19ee593-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:05.606428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.606392 6389 generic.go:356] "Generic (PLEG): container finished" podID="2d0ce741-6676-438e-933b-47a0a19ee593" containerID="9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95" exitCode=0 Nov 03 14:38:05.606428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.606428 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" event={"ID":"2d0ce741-6676-438e-933b-47a0a19ee593","Type":"ContainerDied","Data":"9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95"} Nov 03 14:38:05.606730 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.606481 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" event={"ID":"2d0ce741-6676-438e-933b-47a0a19ee593","Type":"ContainerDied","Data":"5ca19d9726d5dbd25c4d9acdab747f185d208b55b2158f1774396ea34160af93"} Nov 03 14:38:05.606730 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.606508 6389 scope.go:122] "RemoveContainer" containerID="9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95" Nov 03 14:38:05.606919 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.606884 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc" Nov 03 14:38:05.713481 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.713441 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc"] Nov 03 14:38:05.716464 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.716430 6389 scope.go:122] "RemoveContainer" containerID="9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95" Nov 03 14:38:05.717299 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:05.717065 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95\": container with ID starting with 9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95 not found: ID does not exist" containerID="9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95" Nov 03 14:38:05.717299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.717108 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95"} err="failed to get container status \"9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95\": rpc error: code = NotFound desc = could not find container \"9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95\": container with ID starting with 9b24bafd47a47f8532f6523e53a00c97c5fd9a09f29c73d43de758a9efbfcd95 not found: ID does not exist" Nov 03 14:38:05.757646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:05.751302 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1ffaa0-7566569fc9-w98rc"] Nov 03 14:38:06.644637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:06.642873 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0ce741-6676-438e-933b-47a0a19ee593" path="/var/lib/kubelet/pods/2d0ce741-6676-438e-933b-47a0a19ee593/volumes" Nov 03 14:38:07.022030 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:07.021999 6389 scope.go:122] "RemoveContainer" containerID="78208333ea53d9195e7016e168d7d3f10353db9d028157238d9c725f50c17519" Nov 03 14:38:12.337383 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:12.337335 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" podUID="0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:38:13.123006 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.122970 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4"] Nov 03 14:38:13.123898 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.123874 6389 cpu_manager.go:414] "containerMap: removing container" podUID="151733fd-8ae3-4401-870c-8384e18d768b" containerName="main" Nov 03 14:38:13.123898 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.123900 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151733fd-8ae3-4401-870c-8384e18d768b" containerName="main" Nov 03 14:38:13.124121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.123941 6389 cpu_manager.go:414] "containerMap: removing container" podUID="2d0ce741-6676-438e-933b-47a0a19ee593" containerName="maas-api" Nov 03 14:38:13.124121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.123949 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0ce741-6676-438e-933b-47a0a19ee593" containerName="maas-api" Nov 03 14:38:13.124121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.123983 6389 cpu_manager.go:414] "containerMap: removing container" podUID="76e983d7-ca69-44d1-857c-65eb03ec5e6a" containerName="main" Nov 03 14:38:13.124121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.123992 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e983d7-ca69-44d1-857c-65eb03ec5e6a" containerName="main" Nov 03 14:38:13.124121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.124008 6389 cpu_manager.go:414] "containerMap: removing container" podUID="c039791a-d490-415d-9c6f-467a6ca9552a" containerName="payload-processing" Nov 03 14:38:13.124121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.124015 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c039791a-d490-415d-9c6f-467a6ca9552a" containerName="payload-processing" Nov 03 14:38:13.124339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.124191 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0ce741-6676-438e-933b-47a0a19ee593" containerName="maas-api" Nov 03 14:38:13.124339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.124208 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="151733fd-8ae3-4401-870c-8384e18d768b" containerName="main" Nov 03 14:38:13.124339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.124223 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e983d7-ca69-44d1-857c-65eb03ec5e6a" containerName="main" Nov 03 14:38:13.124339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.124234 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="09311533-6310-4062-b1a2-1167c4e4bbc2" containerName="istio-proxy" Nov 03 14:38:13.124339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.124246 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="c039791a-d490-415d-9c6f-467a6ca9552a" containerName="payload-processing" Nov 03 14:38:13.130084 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.129765 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.138904 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.138875 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4"] Nov 03 14:38:13.239990 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.239951 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt"] Nov 03 14:38:13.240251 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.240203 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" podUID="86cddad1-9fbc-482a-9347-b8f02643e2ff" containerName="istio-proxy" containerID="cri-o://3defcf887dd72b1c4a1bc84198db677b3c9f8cc923db275220462e8e38942686" gracePeriod=30 Nov 03 14:38:13.241598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.241465 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.241598 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.241523 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.241825 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.241672 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305d8254-7294-4a52-afa2-0887d742f8f9-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.241825 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.241714 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305d8254-7294-4a52-afa2-0887d742f8f9-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.241825 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.241743 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.241825 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.241789 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.242141 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.241830 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czbbl\" (UniqueName: \"kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-kube-api-access-czbbl\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.242141 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.241895 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.242141 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.241962 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.244175 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.244147 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl"] Nov 03 14:38:13.244428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.244380 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" podUID="a413656c-557f-4262-b40c-a0cf1e193ac6" containerName="istio-proxy" containerID="cri-o://19022acb1d19eb1fe2c878bc10a0dcd231f68bf4e745d5701c89efcaf1a93087" gracePeriod=30 Nov 03 14:38:13.354630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.346201 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.354630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.346266 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czbbl\" (UniqueName: \"kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-kube-api-access-czbbl\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.354630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.346337 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.354630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.346402 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.354630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.346473 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.354630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.346500 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.354630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.346648 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305d8254-7294-4a52-afa2-0887d742f8f9-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.354630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.346695 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305d8254-7294-4a52-afa2-0887d742f8f9-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.354630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.346725 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.354630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.347199 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.354630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.347464 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.400682 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.359800 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.400682 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.360396 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.400682 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.360707 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305d8254-7294-4a52-afa2-0887d742f8f9-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.400682 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.362218 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" podUID="2ce7aee1-fade-4b22-bf65-c3e6e6119db8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:38:13.400682 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.388144 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.504423 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.458387 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.505115 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.463477 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czbbl\" (UniqueName: \"kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-kube-api-access-czbbl\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.505300 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.502574 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305d8254-7294-4a52-afa2-0887d742f8f9-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:13.749431 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:13.749393 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:14.070644 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:14.070591 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4"] Nov 03 14:38:14.071906 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:38:14.071881 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305d8254_7294_4a52_afa2_0887d742f8f9.slice/crio-eab22afc4082e9c2b9de6721dc4446506cffce546d6d4bf24600f4a4c35cbbb2 WatchSource:0}: Error finding container eab22afc4082e9c2b9de6721dc4446506cffce546d6d4bf24600f4a4c35cbbb2: Status 404 returned error can't find the container with id eab22afc4082e9c2b9de6721dc4446506cffce546d6d4bf24600f4a4c35cbbb2 Nov 03 14:38:14.074393 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:14.074360 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:38:14.074465 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:14.074452 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:38:14.074504 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:14.074487 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:38:14.712739 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:14.712700 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" event={"ID":"305d8254-7294-4a52-afa2-0887d742f8f9","Type":"ContainerStarted","Data":"cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210"} Nov 03 14:38:14.713132 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:14.712767 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" event={"ID":"305d8254-7294-4a52-afa2-0887d742f8f9","Type":"ContainerStarted","Data":"eab22afc4082e9c2b9de6721dc4446506cffce546d6d4bf24600f4a4c35cbbb2"} Nov 03 14:38:14.749743 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:14.749688 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:14.789474 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:14.789415 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" podStartSLOduration=1.789395882 podStartE2EDuration="1.789395882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:38:14.787340753 +0000 UTC m=+1570.345977238" watchObservedRunningTime="2026-11-03 14:38:14.789395882 +0000 UTC m=+1570.348032367" Nov 03 14:38:15.751933 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:15.751896 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 14:38:15.752323 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:15.751956 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 14:38:16.753073 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:16.753021 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 14:38:16.753711 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:16.753676 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 14:38:16.784967 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:16.784931 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg"] Nov 03 14:38:16.785984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:16.785950 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" podUID="3c39da63-41cb-41ac-b65a-a4f51ffc9ba2" containerName="maas-api" containerID="cri-o://172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b" gracePeriod=30 Nov 03 14:38:17.161675 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.139117 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:38:17.186694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.186234 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-kube-api-access-2bh5x\" (UniqueName: \"kubernetes.io/projected/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-kube-api-access-2bh5x\") pod \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " Nov 03 14:38:17.186694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.186394 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-metrics-certs\") pod \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " Nov 03 14:38:17.186694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.186495 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-maas-api-tls\") pod \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " Nov 03 14:38:17.186694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.186649 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-openshift-service-ca\") pod \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\" (UID: \"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2\") " Nov 03 14:38:17.187884 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.187855 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-openshift-service-ca" pod "3c39da63-41cb-41ac-b65a-a4f51ffc9ba2" (UID: "3c39da63-41cb-41ac-b65a-a4f51ffc9ba2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:17.202277 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.202240 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-maas-api-tls" pod "3c39da63-41cb-41ac-b65a-a4f51ffc9ba2" (UID: "3c39da63-41cb-41ac-b65a-a4f51ffc9ba2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:38:17.202444 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.202364 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-metrics-certs" pod "3c39da63-41cb-41ac-b65a-a4f51ffc9ba2" (UID: "3c39da63-41cb-41ac-b65a-a4f51ffc9ba2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:38:17.203057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.203018 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-kube-api-access-2bh5x" pod "3c39da63-41cb-41ac-b65a-a4f51ffc9ba2" (UID: "3c39da63-41cb-41ac-b65a-a4f51ffc9ba2"). InnerVolumeSpecName "kube-api-access-2bh5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:17.288799 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.288770 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bh5x\" (UniqueName: \"kubernetes.io/projected/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-kube-api-access-2bh5x\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:17.288799 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.288796 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:17.288799 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.288805 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:17.289040 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.288814 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:17.750133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.750105 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 14:38:17.750315 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.750156 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 14:38:17.756068 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.756039 6389 generic.go:356] "Generic (PLEG): container finished" podID="3c39da63-41cb-41ac-b65a-a4f51ffc9ba2" containerID="172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b" exitCode=0 Nov 03 14:38:17.756538 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.756102 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" Nov 03 14:38:17.756664 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.756125 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" event={"ID":"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2","Type":"ContainerDied","Data":"172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b"} Nov 03 14:38:17.756721 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.756680 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg" event={"ID":"3c39da63-41cb-41ac-b65a-a4f51ffc9ba2","Type":"ContainerDied","Data":"8ba6579505af409fd966786b1b283b8f3d857942ca34d5ba729d4d2d539c96b9"} Nov 03 14:38:17.756721 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.756705 6389 scope.go:122] "RemoveContainer" containerID="172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b" Nov 03 14:38:17.768573 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.768552 6389 scope.go:122] "RemoveContainer" containerID="172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b" Nov 03 14:38:17.768918 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:17.768886 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b\": container with ID starting with 172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b not found: ID does not exist" containerID="172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b" Nov 03 14:38:17.768959 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.768926 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b"} err="failed to get container status \"172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b\": rpc error: code = NotFound desc = could not find container \"172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b\": container with ID starting with 172398bbf112e8353af5215e4d27c55a68737c4745431672cdd03fad08f58e2b not found: ID does not exist" Nov 03 14:38:17.845859 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.845826 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg"] Nov 03 14:38:17.850514 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:17.850485 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-86c2e2-65d9849df7-wtjlg"] Nov 03 14:38:18.801052 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.783418 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 14:38:18.801052 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.783485 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 14:38:18.801052 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.797381 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c39da63-41cb-41ac-b65a-a4f51ffc9ba2" path="/var/lib/kubelet/pods/3c39da63-41cb-41ac-b65a-a4f51ffc9ba2/volumes" Nov 03 14:38:18.808044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.807860 6389 generic.go:356] "Generic (PLEG): container finished" podID="86cddad1-9fbc-482a-9347-b8f02643e2ff" containerID="3defcf887dd72b1c4a1bc84198db677b3c9f8cc923db275220462e8e38942686" exitCode=0 Nov 03 14:38:18.808044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.807960 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" event={"ID":"86cddad1-9fbc-482a-9347-b8f02643e2ff","Type":"ContainerDied","Data":"3defcf887dd72b1c4a1bc84198db677b3c9f8cc923db275220462e8e38942686"} Nov 03 14:38:18.812222 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.809924 6389 generic.go:356] "Generic (PLEG): container finished" podID="a413656c-557f-4262-b40c-a0cf1e193ac6" containerID="19022acb1d19eb1fe2c878bc10a0dcd231f68bf4e745d5701c89efcaf1a93087" exitCode=0 Nov 03 14:38:18.812222 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.810035 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" event={"ID":"a413656c-557f-4262-b40c-a0cf1e193ac6","Type":"ContainerDied","Data":"19022acb1d19eb1fe2c878bc10a0dcd231f68bf4e745d5701c89efcaf1a93087"} Nov 03 14:38:18.816738 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.815560 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:38:18.818067 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.817943 6389 generic.go:356] "Generic (PLEG): container finished" podID="2ce7aee1-fade-4b22-bf65-c3e6e6119db8" containerID="152948f06ef3372801f448168e4cc34678ff250bbe2123cfc268fca4ab80e0dc" exitCode=0 Nov 03 14:38:18.818067 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.818004 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" event={"ID":"2ce7aee1-fade-4b22-bf65-c3e6e6119db8","Type":"ContainerDied","Data":"152948f06ef3372801f448168e4cc34678ff250bbe2123cfc268fca4ab80e0dc"} Nov 03 14:38:18.917449 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.917422 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-envoy\") pod \"a413656c-557f-4262-b40c-a0cf1e193ac6\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " Nov 03 14:38:18.918139 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.918118 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a413656c-557f-4262-b40c-a0cf1e193ac6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a413656c-557f-4262-b40c-a0cf1e193ac6-istiod-ca-cert\") pod \"a413656c-557f-4262-b40c-a0cf1e193ac6\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " Nov 03 14:38:18.918707 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.918689 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-certs\") pod \"a413656c-557f-4262-b40c-a0cf1e193ac6\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " Nov 03 14:38:18.922808 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.922790 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-data\") pod \"a413656c-557f-4262-b40c-a0cf1e193ac6\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " Nov 03 14:38:18.923285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.923268 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-socket\") pod \"a413656c-557f-4262-b40c-a0cf1e193ac6\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " Nov 03 14:38:18.923664 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.923647 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-podinfo\") pod \"a413656c-557f-4262-b40c-a0cf1e193ac6\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " Nov 03 14:38:18.924165 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.924149 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-kube-api-access-kxpsf\" (UniqueName: \"kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-kube-api-access-kxpsf\") pod \"a413656c-557f-4262-b40c-a0cf1e193ac6\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " Nov 03 14:38:18.924717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.918637 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a413656c-557f-4262-b40c-a0cf1e193ac6-istiod-ca-cert" pod "a413656c-557f-4262-b40c-a0cf1e193ac6" (UID: "a413656c-557f-4262-b40c-a0cf1e193ac6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:18.924717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.922691 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-certs" pod "a413656c-557f-4262-b40c-a0cf1e193ac6" (UID: "a413656c-557f-4262-b40c-a0cf1e193ac6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:18.924717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.923118 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-data" pod "a413656c-557f-4262-b40c-a0cf1e193ac6" (UID: "a413656c-557f-4262-b40c-a0cf1e193ac6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:18.924717 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.923573 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-socket" pod "a413656c-557f-4262-b40c-a0cf1e193ac6" (UID: "a413656c-557f-4262-b40c-a0cf1e193ac6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:18.925290 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.924983 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-credential-socket" pod "a413656c-557f-4262-b40c-a0cf1e193ac6" (UID: "a413656c-557f-4262-b40c-a0cf1e193ac6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:18.926516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.925546 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-credential-socket\") pod \"a413656c-557f-4262-b40c-a0cf1e193ac6\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " Nov 03 14:38:18.926516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.925692 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-token\" (UniqueName: \"kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-token\") pod \"a413656c-557f-4262-b40c-a0cf1e193ac6\" (UID: \"a413656c-557f-4262-b40c-a0cf1e193ac6\") " Nov 03 14:38:18.926516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.926226 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a413656c-557f-4262-b40c-a0cf1e193ac6-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:18.926516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.926244 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:18.926516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.926256 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:18.926516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.926268 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:18.926516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:18.926282 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.264680 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.193707 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-podinfo" pod "a413656c-557f-4262-b40c-a0cf1e193ac6" (UID: "a413656c-557f-4262-b40c-a0cf1e193ac6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:19.264680 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.194125 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-envoy" pod "a413656c-557f-4262-b40c-a0cf1e193ac6" (UID: "a413656c-557f-4262-b40c-a0cf1e193ac6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.264680 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.194259 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-token" pod "a413656c-557f-4262-b40c-a0cf1e193ac6" (UID: "a413656c-557f-4262-b40c-a0cf1e193ac6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:19.264680 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.194358 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-kube-api-access-kxpsf" pod "a413656c-557f-4262-b40c-a0cf1e193ac6" (UID: "a413656c-557f-4262-b40c-a0cf1e193ac6"). InnerVolumeSpecName "kube-api-access-kxpsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:19.345924 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.324643 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxpsf\" (UniqueName: \"kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-kube-api-access-kxpsf\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.345924 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.324686 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.345924 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.324703 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.345924 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.324716 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a413656c-557f-4262-b40c-a0cf1e193ac6-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.381008 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.377286 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:38:19.427674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.426452 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-certs\") pod \"86cddad1-9fbc-482a-9347-b8f02643e2ff\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " Nov 03 14:38:19.427674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.426534 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-podinfo\") pod \"86cddad1-9fbc-482a-9347-b8f02643e2ff\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " Nov 03 14:38:19.427674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.426592 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-envoy\") pod \"86cddad1-9fbc-482a-9347-b8f02643e2ff\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " Nov 03 14:38:19.427674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.426725 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-token\" (UniqueName: \"kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-token\") pod \"86cddad1-9fbc-482a-9347-b8f02643e2ff\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " Nov 03 14:38:19.427674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.426799 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-data\") pod \"86cddad1-9fbc-482a-9347-b8f02643e2ff\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " Nov 03 14:38:19.427674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.426832 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-credential-socket\") pod \"86cddad1-9fbc-482a-9347-b8f02643e2ff\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " Nov 03 14:38:19.427674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.426886 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-socket\") pod \"86cddad1-9fbc-482a-9347-b8f02643e2ff\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " Nov 03 14:38:19.427674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.426956 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86cddad1-9fbc-482a-9347-b8f02643e2ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cddad1-9fbc-482a-9347-b8f02643e2ff-istiod-ca-cert\") pod \"86cddad1-9fbc-482a-9347-b8f02643e2ff\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " Nov 03 14:38:19.427674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.427024 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-kube-api-access-kqtcp\" (UniqueName: \"kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-kube-api-access-kqtcp\") pod \"86cddad1-9fbc-482a-9347-b8f02643e2ff\" (UID: \"86cddad1-9fbc-482a-9347-b8f02643e2ff\") " Nov 03 14:38:19.490331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.461868 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-certs" pod "86cddad1-9fbc-482a-9347-b8f02643e2ff" (UID: "86cddad1-9fbc-482a-9347-b8f02643e2ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.490331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.468108 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-credential-socket" pod "86cddad1-9fbc-482a-9347-b8f02643e2ff" (UID: "86cddad1-9fbc-482a-9347-b8f02643e2ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.490331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.468307 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-data" pod "86cddad1-9fbc-482a-9347-b8f02643e2ff" (UID: "86cddad1-9fbc-482a-9347-b8f02643e2ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.490331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.468482 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-socket" pod "86cddad1-9fbc-482a-9347-b8f02643e2ff" (UID: "86cddad1-9fbc-482a-9347-b8f02643e2ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.566646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.559814 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.566646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.559849 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.566646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.559865 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.566646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.559881 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.584466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.579753 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86cddad1-9fbc-482a-9347-b8f02643e2ff-istiod-ca-cert" pod "86cddad1-9fbc-482a-9347-b8f02643e2ff" (UID: "86cddad1-9fbc-482a-9347-b8f02643e2ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:19.617665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.613740 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-podinfo" pod "86cddad1-9fbc-482a-9347-b8f02643e2ff" (UID: "86cddad1-9fbc-482a-9347-b8f02643e2ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:19.617665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.613770 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-token" pod "86cddad1-9fbc-482a-9347-b8f02643e2ff" (UID: "86cddad1-9fbc-482a-9347-b8f02643e2ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:19.682687 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.678309 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-kube-api-access-kqtcp" pod "86cddad1-9fbc-482a-9347-b8f02643e2ff" (UID: "86cddad1-9fbc-482a-9347-b8f02643e2ff"). InnerVolumeSpecName "kube-api-access-kqtcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:19.695623 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.693477 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-envoy" pod "86cddad1-9fbc-482a-9347-b8f02643e2ff" (UID: "86cddad1-9fbc-482a-9347-b8f02643e2ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.695623 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.695031 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.695623 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.695058 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.695623 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.695076 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.695623 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.695089 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cddad1-9fbc-482a-9347-b8f02643e2ff-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.695623 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.695104 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqtcp\" (UniqueName: \"kubernetes.io/projected/86cddad1-9fbc-482a-9347-b8f02643e2ff-kube-api-access-kqtcp\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.766805 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.765733 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 14:38:19.766805 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.765807 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 14:38:19.846713 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.845724 6389 generic.go:356] "Generic (PLEG): container finished" podID="0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6" containerID="823359b8ac65b6ff416da9649eaa08ed39c036abaa263d6d8ad7d8281e9b5b05" exitCode=0 Nov 03 14:38:19.846713 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.845916 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" event={"ID":"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6","Type":"ContainerDied","Data":"823359b8ac65b6ff416da9649eaa08ed39c036abaa263d6d8ad7d8281e9b5b05"} Nov 03 14:38:19.878543 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.867949 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" event={"ID":"86cddad1-9fbc-482a-9347-b8f02643e2ff","Type":"ContainerDied","Data":"785a94d610d87fef23100a50206311cc1a4dd984a099575950360484b670c2ae"} Nov 03 14:38:19.878543 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.868022 6389 scope.go:122] "RemoveContainer" containerID="3defcf887dd72b1c4a1bc84198db677b3c9f8cc923db275220462e8e38942686" Nov 03 14:38:19.878543 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.868146 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt" Nov 03 14:38:19.890758 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.890706 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" event={"ID":"a413656c-557f-4262-b40c-a0cf1e193ac6","Type":"ContainerDied","Data":"6b525bee9be679db26fa912f7c57f88b38308148ec5335773e468383ee832456"} Nov 03 14:38:19.890942 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.890830 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl" Nov 03 14:38:19.900207 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.900182 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" Nov 03 14:38:19.904708 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.904686 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" Nov 03 14:38:19.911240 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:19.911219 6389 scope.go:122] "RemoveContainer" containerID="19022acb1d19eb1fe2c878bc10a0dcd231f68bf4e745d5701c89efcaf1a93087" Nov 03 14:38:20.000090 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.000041 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-kube-api-access-ncnz5\" (UniqueName: \"kubernetes.io/projected/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-kube-api-access-ncnz5\") pod \"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6\" (UID: \"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6\") " Nov 03 14:38:20.000090 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.000075 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-kube-api-access-5989m\" (UniqueName: \"kubernetes.io/projected/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-kube-api-access-5989m\") pod \"2ce7aee1-fade-4b22-bf65-c3e6e6119db8\" (UID: \"2ce7aee1-fade-4b22-bf65-c3e6e6119db8\") " Nov 03 14:38:20.000420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.000166 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl"] Nov 03 14:38:20.000420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.000193 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-plugins-config-volume\") pod \"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6\" (UID: \"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6\") " Nov 03 14:38:20.000420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.000227 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-plugins-config-volume\") pod \"2ce7aee1-fade-4b22-bf65-c3e6e6119db8\" (UID: \"2ce7aee1-fade-4b22-bf65-c3e6e6119db8\") " Nov 03 14:38:20.000887 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.000856 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-plugins-config-volume" pod "2ce7aee1-fade-4b22-bf65-c3e6e6119db8" (UID: "2ce7aee1-fade-4b22-bf65-c3e6e6119db8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:20.001442 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.001295 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-plugins-config-volume" pod "0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6" (UID: "0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:20.002635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.002581 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-kube-api-access-ncnz5" pod "0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6" (UID: "0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6"). InnerVolumeSpecName "kube-api-access-ncnz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:20.002888 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.002870 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-kube-api-access-5989m" pod "2ce7aee1-fade-4b22-bf65-c3e6e6119db8" (UID: "2ce7aee1-fade-4b22-bf65-c3e6e6119db8"). InnerVolumeSpecName "kube-api-access-5989m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:20.012508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.012481 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-69hsl"] Nov 03 14:38:20.106601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.104408 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt"] Nov 03 14:38:20.106601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.105808 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:20.106601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.105832 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:20.106601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.105849 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncnz5\" (UniqueName: \"kubernetes.io/projected/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6-kube-api-access-ncnz5\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:20.106601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.105868 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5989m\" (UniqueName: \"kubernetes.io/projected/2ce7aee1-fade-4b22-bf65-c3e6e6119db8-kube-api-access-5989m\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:20.110042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.110011 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-x9tzt"] Nov 03 14:38:20.622884 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.622847 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86cddad1-9fbc-482a-9347-b8f02643e2ff" path="/var/lib/kubelet/pods/86cddad1-9fbc-482a-9347-b8f02643e2ff/volumes" Nov 03 14:38:20.623595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.623576 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a413656c-557f-4262-b40c-a0cf1e193ac6" path="/var/lib/kubelet/pods/a413656c-557f-4262-b40c-a0cf1e193ac6/volumes" Nov 03 14:38:20.749905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.749872 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 14:38:20.750070 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.749938 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 14:38:20.901499 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.901409 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" Nov 03 14:38:20.901499 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.901426 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw" event={"ID":"0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6","Type":"ContainerDied","Data":"400a3e71c6ee9e8a9a6a34196dc667dc61ff5b1df11fe96512fe41ede293d0e0"} Nov 03 14:38:20.901499 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.901489 6389 scope.go:122] "RemoveContainer" containerID="823359b8ac65b6ff416da9649eaa08ed39c036abaa263d6d8ad7d8281e9b5b05" Nov 03 14:38:20.903206 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.903077 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" event={"ID":"2ce7aee1-fade-4b22-bf65-c3e6e6119db8","Type":"ContainerDied","Data":"bf725c1b75875db317d2c5ab0ecc08bf93d14ef580774bc7ec74dbab423d6d7a"} Nov 03 14:38:20.903206 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.903140 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88" Nov 03 14:38:20.911564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.911546 6389 scope.go:122] "RemoveContainer" containerID="152948f06ef3372801f448168e4cc34678ff250bbe2123cfc268fca4ab80e0dc" Nov 03 14:38:20.959757 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.959724 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88"] Nov 03 14:38:20.967006 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:20.966973 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1ffaa0-7db5bbf6b7-pjw88"] Nov 03 14:38:21.011372 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:21.011338 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw"] Nov 03 14:38:21.016779 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:21.016758 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1ffaa0-7b44764f75-22szw"] Nov 03 14:38:22.632009 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:22.631976 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6" path="/var/lib/kubelet/pods/0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6/volumes" Nov 03 14:38:22.632490 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:22.632475 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce7aee1-fade-4b22-bf65-c3e6e6119db8" path="/var/lib/kubelet/pods/2ce7aee1-fade-4b22-bf65-c3e6e6119db8/volumes" Nov 03 14:38:22.751416 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:22.751378 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:38:22.751646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:22.751436 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:38:23.808815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:23.796717 6389 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:38:23.808815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:23.796874 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:38:24.314653 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.304965 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:24.314653 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.306588 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:24.314653 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.310036 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:24.837129 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.837098 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8"] Nov 03 14:38:24.837980 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.837953 6389 cpu_manager.go:414] "containerMap: removing container" podUID="0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6" containerName="payload-pre-processing" Nov 03 14:38:24.837980 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.837978 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6" containerName="payload-pre-processing" Nov 03 14:38:24.838200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.837997 6389 cpu_manager.go:414] "containerMap: removing container" podUID="86cddad1-9fbc-482a-9347-b8f02643e2ff" containerName="istio-proxy" Nov 03 14:38:24.838200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.838005 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86cddad1-9fbc-482a-9347-b8f02643e2ff" containerName="istio-proxy" Nov 03 14:38:24.838200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.838023 6389 cpu_manager.go:414] "containerMap: removing container" podUID="a413656c-557f-4262-b40c-a0cf1e193ac6" containerName="istio-proxy" Nov 03 14:38:24.838200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.838030 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a413656c-557f-4262-b40c-a0cf1e193ac6" containerName="istio-proxy" Nov 03 14:38:24.838200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.838049 6389 cpu_manager.go:414] "containerMap: removing container" podUID="2ce7aee1-fade-4b22-bf65-c3e6e6119db8" containerName="payload-processing" Nov 03 14:38:24.838200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.838054 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce7aee1-fade-4b22-bf65-c3e6e6119db8" containerName="payload-processing" Nov 03 14:38:24.838200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.838064 6389 cpu_manager.go:414] "containerMap: removing container" podUID="3c39da63-41cb-41ac-b65a-a4f51ffc9ba2" containerName="maas-api" Nov 03 14:38:24.838200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.838068 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c39da63-41cb-41ac-b65a-a4f51ffc9ba2" containerName="maas-api" Nov 03 14:38:24.838200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.838167 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="a413656c-557f-4262-b40c-a0cf1e193ac6" containerName="istio-proxy" Nov 03 14:38:24.838200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.838179 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="86cddad1-9fbc-482a-9347-b8f02643e2ff" containerName="istio-proxy" Nov 03 14:38:24.838200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.838189 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce7aee1-fade-4b22-bf65-c3e6e6119db8" containerName="payload-processing" Nov 03 14:38:24.838200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.838198 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c39da63-41cb-41ac-b65a-a4f51ffc9ba2" containerName="maas-api" Nov 03 14:38:24.838882 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.838213 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa8a4f0-30e5-4c0b-9d31-95911c3f0de6" containerName="payload-pre-processing" Nov 03 14:38:24.841726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.841703 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:24.861201 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.861173 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-e2e--e9e6f325-dockercfg-vdscv\"" Nov 03 14:38:24.861504 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.861301 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"e2e-cfl-d4022b-provider-cred\"" Nov 03 14:38:24.865494 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.865466 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8"] Nov 03 14:38:24.872428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.872379 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"kube-root-ca.crt\"" Nov 03 14:38:24.872572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.872425 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-331c9e\"" Nov 03 14:38:24.872572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.872429 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"openshift-service-ca.crt\"" Nov 03 14:38:24.872572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.872558 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"routing-overlay\"" Nov 03 14:38:24.950788 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.950741 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:24.950941 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.950811 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dkzg\" (UniqueName: \"kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-kube-api-access-6dkzg\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:24.951035 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.951011 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:24.951161 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:24.951139 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:25.052368 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.052324 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:25.052555 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.052390 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dkzg\" (UniqueName: \"kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-kube-api-access-6dkzg\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:25.052702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.052678 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:25.053237 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.053206 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:25.053418 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.053205 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:25.054299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.054270 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:25.055364 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.055340 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:25.074316 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.074291 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dkzg\" (UniqueName: \"kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-kube-api-access-6dkzg\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:25.158229 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.158136 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:25.472044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.472014 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8"] Nov 03 14:38:25.473515 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:38:25.473489 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d86ec84_3f92_45a7_a553_4f20322ac5be.slice/crio-16fdf9b514b40e889716bc2c1d61efa5d1013091ae365b3d148dd99c9183df3a WatchSource:0}: Error finding container 16fdf9b514b40e889716bc2c1d61efa5d1013091ae365b3d148dd99c9183df3a: Status 404 returned error can't find the container with id 16fdf9b514b40e889716bc2c1d61efa5d1013091ae365b3d148dd99c9183df3a Nov 03 14:38:25.951147 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.951112 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" event={"ID":"7d86ec84-3f92-45a7-a553-4f20322ac5be","Type":"ContainerStarted","Data":"8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9"} Nov 03 14:38:25.951700 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.951214 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" event={"ID":"7d86ec84-3f92-45a7-a553-4f20322ac5be","Type":"ContainerStarted","Data":"16fdf9b514b40e889716bc2c1d61efa5d1013091ae365b3d148dd99c9183df3a"} Nov 03 14:38:25.951700 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.951250 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:25.995722 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:25.995671 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" podStartSLOduration=1.995657416 podStartE2EDuration="1.995657416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:38:25.993745832 +0000 UTC m=+1581.552382315" watchObservedRunningTime="2026-11-03 14:38:25.995657416 +0000 UTC m=+1581.554293899" Nov 03 14:38:26.156936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:26.150772 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2"] Nov 03 14:38:26.156936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:26.151101 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" containerID="cri-o://6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b" gracePeriod=30 Nov 03 14:38:26.156936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:26.156425 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd"] Nov 03 14:38:26.156936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:26.156656 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" podUID="30003229-866c-4a72-be45-37c3134e3159" containerName="istio-proxy" containerID="cri-o://78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07" gracePeriod=30 Nov 03 14:38:26.174080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:26.174041 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4"] Nov 03 14:38:26.178988 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:26.178963 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4"] Nov 03 14:38:26.179179 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:26.179149 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" podUID="5c43724b-ae54-408e-8e4d-b307cd947d9c" containerName="istio-proxy" containerID="cri-o://d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da" gracePeriod=30 Nov 03 14:38:26.957059 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:26.957019 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" containerName="istio-proxy" containerID="cri-o://cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210" gracePeriod=30 Nov 03 14:38:28.374293 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:28.374248 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:38:31.450225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.450194 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:38:31.504799 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.504776 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:38:31.508078 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.508059 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:38:31.560632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.553118 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-podinfo\") pod \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " Nov 03 14:38:31.560632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.553250 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-socket\") pod \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " Nov 03 14:38:31.560632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.553286 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istiod-ca-cert\") pod \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " Nov 03 14:38:31.567114 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.567081 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-credential-socket\") pod \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " Nov 03 14:38:31.567310 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.567161 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-certs\") pod \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " Nov 03 14:38:31.567310 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.567235 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-kube-api-access-r2zdw\" (UniqueName: \"kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-kube-api-access-r2zdw\") pod \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " Nov 03 14:38:31.567505 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.567313 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-envoy\") pod \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " Nov 03 14:38:31.567505 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.567363 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-token\" (UniqueName: \"kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-token\") pod \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " Nov 03 14:38:31.567505 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.567424 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-data\") pod \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\" (UID: \"3d0cd10d-efc2-47f3-860c-5db6ed4b38db\") " Nov 03 14:38:31.593064 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.568154 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-certs" pod "3d0cd10d-efc2-47f3-860c-5db6ed4b38db" (UID: "3d0cd10d-efc2-47f3-860c-5db6ed4b38db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.593064 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.568376 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.593064 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.568387 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-data" pod "3d0cd10d-efc2-47f3-860c-5db6ed4b38db" (UID: "3d0cd10d-efc2-47f3-860c-5db6ed4b38db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.593064 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.569032 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istiod-ca-cert" pod "3d0cd10d-efc2-47f3-860c-5db6ed4b38db" (UID: "3d0cd10d-efc2-47f3-860c-5db6ed4b38db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:31.593064 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.569388 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-credential-socket" pod "3d0cd10d-efc2-47f3-860c-5db6ed4b38db" (UID: "3d0cd10d-efc2-47f3-860c-5db6ed4b38db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.593064 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.574313 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-token" pod "3d0cd10d-efc2-47f3-860c-5db6ed4b38db" (UID: "3d0cd10d-efc2-47f3-860c-5db6ed4b38db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:31.593064 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.575729 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-envoy" pod "3d0cd10d-efc2-47f3-860c-5db6ed4b38db" (UID: "3d0cd10d-efc2-47f3-860c-5db6ed4b38db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.593064 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.586763 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-podinfo" pod "3d0cd10d-efc2-47f3-860c-5db6ed4b38db" (UID: "3d0cd10d-efc2-47f3-860c-5db6ed4b38db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:31.593064 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.592993 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-socket" pod "3d0cd10d-efc2-47f3-860c-5db6ed4b38db" (UID: "3d0cd10d-efc2-47f3-860c-5db6ed4b38db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.601792 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.601752 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-kube-api-access-r2zdw" pod "3d0cd10d-efc2-47f3-860c-5db6ed4b38db" (UID: "3d0cd10d-efc2-47f3-860c-5db6ed4b38db"). InnerVolumeSpecName "kube-api-access-r2zdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:31.668827 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.668795 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-token\" (UniqueName: \"kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-token\") pod \"5c43724b-ae54-408e-8e4d-b307cd947d9c\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " Nov 03 14:38:31.669033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.668867 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30003229-866c-4a72-be45-37c3134e3159-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30003229-866c-4a72-be45-37c3134e3159-istio-podinfo\") pod \"30003229-866c-4a72-be45-37c3134e3159\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " Nov 03 14:38:31.669033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.668910 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-certs\") pod \"5c43724b-ae54-408e-8e4d-b307cd947d9c\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " Nov 03 14:38:31.669033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.668944 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-socket\") pod \"30003229-866c-4a72-be45-37c3134e3159\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " Nov 03 14:38:31.669033 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.668998 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-envoy\") pod \"5c43724b-ae54-408e-8e4d-b307cd947d9c\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " Nov 03 14:38:31.669250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669035 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-kube-api-access-47cj8\" (UniqueName: \"kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-kube-api-access-47cj8\") pod \"5c43724b-ae54-408e-8e4d-b307cd947d9c\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " Nov 03 14:38:31.669250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669087 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-envoy\") pod \"30003229-866c-4a72-be45-37c3134e3159\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " Nov 03 14:38:31.669250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669131 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-podinfo\") pod \"5c43724b-ae54-408e-8e4d-b307cd947d9c\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " Nov 03 14:38:31.669250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669160 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-credential-socket\") pod \"5c43724b-ae54-408e-8e4d-b307cd947d9c\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " Nov 03 14:38:31.669250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669230 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-socket\") pod \"5c43724b-ae54-408e-8e4d-b307cd947d9c\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " Nov 03 14:38:31.669491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669268 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-kube-api-access-7zzkr\" (UniqueName: \"kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-kube-api-access-7zzkr\") pod \"30003229-866c-4a72-be45-37c3134e3159\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " Nov 03 14:38:31.669491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669308 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-data\") pod \"30003229-866c-4a72-be45-37c3134e3159\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " Nov 03 14:38:31.669491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669372 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30003229-866c-4a72-be45-37c3134e3159-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30003229-866c-4a72-be45-37c3134e3159-istiod-ca-cert\") pod \"30003229-866c-4a72-be45-37c3134e3159\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " Nov 03 14:38:31.669491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669414 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c43724b-ae54-408e-8e4d-b307cd947d9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c43724b-ae54-408e-8e4d-b307cd947d9c-istiod-ca-cert\") pod \"5c43724b-ae54-408e-8e4d-b307cd947d9c\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " Nov 03 14:38:31.669491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669467 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-credential-socket\") pod \"30003229-866c-4a72-be45-37c3134e3159\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " Nov 03 14:38:31.669771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669550 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-istio-token\" (UniqueName: \"kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-istio-token\") pod \"30003229-866c-4a72-be45-37c3134e3159\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " Nov 03 14:38:31.669771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669599 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-certs\") pod \"30003229-866c-4a72-be45-37c3134e3159\" (UID: \"30003229-866c-4a72-be45-37c3134e3159\") " Nov 03 14:38:31.669771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.669719 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-data\") pod \"5c43724b-ae54-408e-8e4d-b307cd947d9c\" (UID: \"5c43724b-ae54-408e-8e4d-b307cd947d9c\") " Nov 03 14:38:31.670786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.670153 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-credential-socket" pod "5c43724b-ae54-408e-8e4d-b307cd947d9c" (UID: "5c43724b-ae54-408e-8e4d-b307cd947d9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.670786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.670284 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.670786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.670333 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2zdw\" (UniqueName: \"kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-kube-api-access-r2zdw\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.670786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.670348 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.670786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.670352 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-data" pod "5c43724b-ae54-408e-8e4d-b307cd947d9c" (UID: "5c43724b-ae54-408e-8e4d-b307cd947d9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.670786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.670363 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.670786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.670403 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.670786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.670416 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.670786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.670430 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.670786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.670445 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.670786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.670462 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d0cd10d-efc2-47f3-860c-5db6ed4b38db-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.671742 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.671714 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-certs" pod "5c43724b-ae54-408e-8e4d-b307cd947d9c" (UID: "5c43724b-ae54-408e-8e4d-b307cd947d9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.673127 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.673100 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-socket" pod "5c43724b-ae54-408e-8e4d-b307cd947d9c" (UID: "5c43724b-ae54-408e-8e4d-b307cd947d9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.673896 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.673719 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-socket" pod "30003229-866c-4a72-be45-37c3134e3159" (UID: "30003229-866c-4a72-be45-37c3134e3159"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.674351 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.674329 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c43724b-ae54-408e-8e4d-b307cd947d9c-istiod-ca-cert" pod "5c43724b-ae54-408e-8e4d-b307cd947d9c" (UID: "5c43724b-ae54-408e-8e4d-b307cd947d9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:31.674556 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.674538 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-data" pod "30003229-866c-4a72-be45-37c3134e3159" (UID: "30003229-866c-4a72-be45-37c3134e3159"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.674857 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.674834 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30003229-866c-4a72-be45-37c3134e3159-istiod-ca-cert" pod "30003229-866c-4a72-be45-37c3134e3159" (UID: "30003229-866c-4a72-be45-37c3134e3159"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:31.675088 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.675062 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-podinfo" pod "5c43724b-ae54-408e-8e4d-b307cd947d9c" (UID: "5c43724b-ae54-408e-8e4d-b307cd947d9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:31.675358 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.675334 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-credential-socket" pod "30003229-866c-4a72-be45-37c3134e3159" (UID: "30003229-866c-4a72-be45-37c3134e3159"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.675572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.675531 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-certs" pod "30003229-866c-4a72-be45-37c3134e3159" (UID: "30003229-866c-4a72-be45-37c3134e3159"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.675572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.675771 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-kube-api-access-47cj8" pod "5c43724b-ae54-408e-8e4d-b307cd947d9c" (UID: "5c43724b-ae54-408e-8e4d-b307cd947d9c"). InnerVolumeSpecName "kube-api-access-47cj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:31.676205 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.676176 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-envoy" pod "30003229-866c-4a72-be45-37c3134e3159" (UID: "30003229-866c-4a72-be45-37c3134e3159"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.677198 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.677177 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-kube-api-access-7zzkr" pod "30003229-866c-4a72-be45-37c3134e3159" (UID: "30003229-866c-4a72-be45-37c3134e3159"). InnerVolumeSpecName "kube-api-access-7zzkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:31.677572 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.677547 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30003229-866c-4a72-be45-37c3134e3159-istio-podinfo" pod "30003229-866c-4a72-be45-37c3134e3159" (UID: "30003229-866c-4a72-be45-37c3134e3159"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:31.677702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.677549 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-envoy" pod "5c43724b-ae54-408e-8e4d-b307cd947d9c" (UID: "5c43724b-ae54-408e-8e4d-b307cd947d9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.677702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.677583 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-token" pod "5c43724b-ae54-408e-8e4d-b307cd947d9c" (UID: "5c43724b-ae54-408e-8e4d-b307cd947d9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:31.677950 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.677929 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-istio-token" pod "30003229-866c-4a72-be45-37c3134e3159" (UID: "30003229-866c-4a72-be45-37c3134e3159"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:31.771595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771508 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771541 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zzkr\" (UniqueName: \"kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-kube-api-access-7zzkr\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771552 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771561 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30003229-866c-4a72-be45-37c3134e3159-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771569 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c43724b-ae54-408e-8e4d-b307cd947d9c-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771577 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771585 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30003229-866c-4a72-be45-37c3134e3159-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771596 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771603 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771642 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771650 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30003229-866c-4a72-be45-37c3134e3159-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771660 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771668 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771675 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771684 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47cj8\" (UniqueName: \"kubernetes.io/projected/5c43724b-ae54-408e-8e4d-b307cd947d9c-kube-api-access-47cj8\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771691 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30003229-866c-4a72-be45-37c3134e3159-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.771978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.771701 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c43724b-ae54-408e-8e4d-b307cd947d9c-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.986547 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.986516 6389 generic.go:356] "Generic (PLEG): container finished" podID="30003229-866c-4a72-be45-37c3134e3159" containerID="78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07" exitCode=0 Nov 03 14:38:31.986688 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.986566 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" Nov 03 14:38:31.986688 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.986620 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" event={"ID":"30003229-866c-4a72-be45-37c3134e3159","Type":"ContainerDied","Data":"78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07"} Nov 03 14:38:31.986803 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.986688 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd" event={"ID":"30003229-866c-4a72-be45-37c3134e3159","Type":"ContainerDied","Data":"128f716c5479d4cbf31abd3c366c9c0b991b87f75fd28408278c9403e809fbf1"} Nov 03 14:38:31.986803 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.986720 6389 scope.go:122] "RemoveContainer" containerID="78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07" Nov 03 14:38:31.988191 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.988167 6389 generic.go:356] "Generic (PLEG): container finished" podID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerID="6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b" exitCode=0 Nov 03 14:38:31.988280 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.988206 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" Nov 03 14:38:31.988341 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.988294 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" event={"ID":"3d0cd10d-efc2-47f3-860c-5db6ed4b38db","Type":"ContainerDied","Data":"6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b"} Nov 03 14:38:31.988386 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.988338 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2" event={"ID":"3d0cd10d-efc2-47f3-860c-5db6ed4b38db","Type":"ContainerDied","Data":"222c06d7b5c9c65685f564a7d4652c8ff43bbb2233af1b7ac73546c105a186f1"} Nov 03 14:38:31.989848 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.989824 6389 generic.go:356] "Generic (PLEG): container finished" podID="5c43724b-ae54-408e-8e4d-b307cd947d9c" containerID="d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da" exitCode=0 Nov 03 14:38:31.989952 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.989865 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" Nov 03 14:38:31.989952 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.989894 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" event={"ID":"5c43724b-ae54-408e-8e4d-b307cd947d9c","Type":"ContainerDied","Data":"d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da"} Nov 03 14:38:31.989952 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:31.989923 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4" event={"ID":"5c43724b-ae54-408e-8e4d-b307cd947d9c","Type":"ContainerDied","Data":"0ba27f8bc43faeee4021df298133c6a1204139386a95851535fb0a28152999e8"} Nov 03 14:38:32.069475 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.069457 6389 scope.go:122] "RemoveContainer" containerID="78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07" Nov 03 14:38:32.069872 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:32.069849 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07\": container with ID starting with 78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07 not found: ID does not exist" containerID="78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07" Nov 03 14:38:32.069978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.069883 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07"} err="failed to get container status \"78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07\": rpc error: code = NotFound desc = could not find container \"78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07\": container with ID starting with 78ce8e517351816db5cf84888bee35c04846d2530bcffdaed7fb3779836c1c07 not found: ID does not exist" Nov 03 14:38:32.069978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.069910 6389 scope.go:122] "RemoveContainer" containerID="6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b" Nov 03 14:38:32.078519 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.078497 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd"] Nov 03 14:38:32.079949 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.079926 6389 scope.go:122] "RemoveContainer" containerID="6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b" Nov 03 14:38:32.080256 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:32.080236 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b\": container with ID starting with 6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b not found: ID does not exist" containerID="6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b" Nov 03 14:38:32.080345 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.080263 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b"} err="failed to get container status \"6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b\": rpc error: code = NotFound desc = could not find container \"6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b\": container with ID starting with 6bd49163522d59ac6f678acd134ea68423b2cb4561dbfbe77d7fae655c887b5b not found: ID does not exist" Nov 03 14:38:32.080345 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.080284 6389 scope.go:122] "RemoveContainer" containerID="d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da" Nov 03 14:38:32.083545 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.083525 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-ssgqd"] Nov 03 14:38:32.091569 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.091548 6389 scope.go:122] "RemoveContainer" containerID="d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da" Nov 03 14:38:32.091896 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:32.091877 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da\": container with ID starting with d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da not found: ID does not exist" containerID="d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da" Nov 03 14:38:32.091968 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.091905 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da"} err="failed to get container status \"d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da\": rpc error: code = NotFound desc = could not find container \"d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da\": container with ID starting with d11bb65695d4b18d758e9aef3bcaafec27d06f13a50987f3c29f600a9f45c8da not found: ID does not exist" Nov 03 14:38:32.127974 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.127940 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4"] Nov 03 14:38:32.137262 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.137241 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-2xzm4"] Nov 03 14:38:32.176184 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.176151 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2"] Nov 03 14:38:32.182037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.182009 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dn5m2"] Nov 03 14:38:32.248915 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.248891 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:32.379255 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379161 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-socket\") pod \"305d8254-7294-4a52-afa2-0887d742f8f9\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " Nov 03 14:38:32.379255 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379212 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-data\") pod \"305d8254-7294-4a52-afa2-0887d742f8f9\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " Nov 03 14:38:32.379255 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379257 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-istio-token\" (UniqueName: \"kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-istio-token\") pod \"305d8254-7294-4a52-afa2-0887d742f8f9\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " Nov 03 14:38:32.379536 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379293 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-kube-api-access-czbbl\" (UniqueName: \"kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-kube-api-access-czbbl\") pod \"305d8254-7294-4a52-afa2-0887d742f8f9\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " Nov 03 14:38:32.379536 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379312 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-envoy\") pod \"305d8254-7294-4a52-afa2-0887d742f8f9\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " Nov 03 14:38:32.379536 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379459 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-certs\") pod \"305d8254-7294-4a52-afa2-0887d742f8f9\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " Nov 03 14:38:32.379536 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379486 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-socket" pod "305d8254-7294-4a52-afa2-0887d742f8f9" (UID: "305d8254-7294-4a52-afa2-0887d742f8f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:32.379536 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379503 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/305d8254-7294-4a52-afa2-0887d742f8f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305d8254-7294-4a52-afa2-0887d742f8f9-istio-podinfo\") pod \"305d8254-7294-4a52-afa2-0887d742f8f9\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " Nov 03 14:38:32.379822 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379539 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-data" pod "305d8254-7294-4a52-afa2-0887d742f8f9" (UID: "305d8254-7294-4a52-afa2-0887d742f8f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:32.379822 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379548 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305d8254-7294-4a52-afa2-0887d742f8f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305d8254-7294-4a52-afa2-0887d742f8f9-istiod-ca-cert\") pod \"305d8254-7294-4a52-afa2-0887d742f8f9\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " Nov 03 14:38:32.379822 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379757 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-certs" pod "305d8254-7294-4a52-afa2-0887d742f8f9" (UID: "305d8254-7294-4a52-afa2-0887d742f8f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:32.379822 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379801 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-credential-socket\") pod \"305d8254-7294-4a52-afa2-0887d742f8f9\" (UID: \"305d8254-7294-4a52-afa2-0887d742f8f9\") " Nov 03 14:38:32.380044 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.379879 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305d8254-7294-4a52-afa2-0887d742f8f9-istiod-ca-cert" pod "305d8254-7294-4a52-afa2-0887d742f8f9" (UID: "305d8254-7294-4a52-afa2-0887d742f8f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:32.380100 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.380038 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-credential-socket" pod "305d8254-7294-4a52-afa2-0887d742f8f9" (UID: "305d8254-7294-4a52-afa2-0887d742f8f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:32.380338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.380319 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:32.380338 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.380340 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:32.380484 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.380353 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:32.380484 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.380365 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:32.380484 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.380375 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305d8254-7294-4a52-afa2-0887d742f8f9-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:32.381764 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.381737 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-istio-token" pod "305d8254-7294-4a52-afa2-0887d742f8f9" (UID: "305d8254-7294-4a52-afa2-0887d742f8f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:32.381849 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.381767 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/305d8254-7294-4a52-afa2-0887d742f8f9-istio-podinfo" pod "305d8254-7294-4a52-afa2-0887d742f8f9" (UID: "305d8254-7294-4a52-afa2-0887d742f8f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:32.382011 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.381991 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-kube-api-access-czbbl" pod "305d8254-7294-4a52-afa2-0887d742f8f9" (UID: "305d8254-7294-4a52-afa2-0887d742f8f9"). InnerVolumeSpecName "kube-api-access-czbbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:32.382011 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.381996 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-envoy" pod "305d8254-7294-4a52-afa2-0887d742f8f9" (UID: "305d8254-7294-4a52-afa2-0887d742f8f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:32.481843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.481805 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:32.481843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.481830 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czbbl\" (UniqueName: \"kubernetes.io/projected/305d8254-7294-4a52-afa2-0887d742f8f9-kube-api-access-czbbl\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:32.481843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.481840 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305d8254-7294-4a52-afa2-0887d742f8f9-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:32.481843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.481850 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305d8254-7294-4a52-afa2-0887d742f8f9-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:32.619024 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.618996 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30003229-866c-4a72-be45-37c3134e3159" path="/var/lib/kubelet/pods/30003229-866c-4a72-be45-37c3134e3159/volumes" Nov 03 14:38:32.619469 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.619456 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" path="/var/lib/kubelet/pods/3d0cd10d-efc2-47f3-860c-5db6ed4b38db/volumes" Nov 03 14:38:32.619898 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:32.619885 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c43724b-ae54-408e-8e4d-b307cd947d9c" path="/var/lib/kubelet/pods/5c43724b-ae54-408e-8e4d-b307cd947d9c/volumes" Nov 03 14:38:33.000118 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.000085 6389 generic.go:356] "Generic (PLEG): container finished" podID="305d8254-7294-4a52-afa2-0887d742f8f9" containerID="cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210" exitCode=0 Nov 03 14:38:33.000304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.000130 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" Nov 03 14:38:33.000304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.000162 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" event={"ID":"305d8254-7294-4a52-afa2-0887d742f8f9","Type":"ContainerDied","Data":"cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210"} Nov 03 14:38:33.000304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.000211 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4" event={"ID":"305d8254-7294-4a52-afa2-0887d742f8f9","Type":"ContainerDied","Data":"eab22afc4082e9c2b9de6721dc4446506cffce546d6d4bf24600f4a4c35cbbb2"} Nov 03 14:38:33.000304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.000232 6389 scope.go:122] "RemoveContainer" containerID="cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210" Nov 03 14:38:33.009465 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.009447 6389 scope.go:122] "RemoveContainer" containerID="cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210" Nov 03 14:38:33.009742 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:33.009723 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210\": container with ID starting with cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210 not found: ID does not exist" containerID="cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210" Nov 03 14:38:33.009831 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.009757 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210"} err="failed to get container status \"cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210\": rpc error: code = NotFound desc = could not find container \"cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210\": container with ID starting with cb6c3920c98b79065465f453b4135982180e5038d3616bd43e0edc5975d25210 not found: ID does not exist" Nov 03 14:38:33.072719 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.072691 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4"] Nov 03 14:38:33.084299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.084272 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-7vns4"] Nov 03 14:38:33.163069 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.163036 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26"] Nov 03 14:38:33.163339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.163309 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" podUID="0c918c5c-994d-4531-ad18-89ee658e2c4e" containerName="maas-api" containerID="cri-o://82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a" gracePeriod=30 Nov 03 14:38:33.497894 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.487379 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:38:33.599459 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.599364 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-metrics-certs\") pod \"0c918c5c-994d-4531-ad18-89ee658e2c4e\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " Nov 03 14:38:33.599459 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.599411 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c918c5c-994d-4531-ad18-89ee658e2c4e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c918c5c-994d-4531-ad18-89ee658e2c4e-openshift-service-ca\") pod \"0c918c5c-994d-4531-ad18-89ee658e2c4e\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " Nov 03 14:38:33.599459 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.599437 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-maas-api-tls\") pod \"0c918c5c-994d-4531-ad18-89ee658e2c4e\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " Nov 03 14:38:33.599797 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.599621 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c918c5c-994d-4531-ad18-89ee658e2c4e-kube-api-access-tt4cl\" (UniqueName: \"kubernetes.io/projected/0c918c5c-994d-4531-ad18-89ee658e2c4e-kube-api-access-tt4cl\") pod \"0c918c5c-994d-4531-ad18-89ee658e2c4e\" (UID: \"0c918c5c-994d-4531-ad18-89ee658e2c4e\") " Nov 03 14:38:33.599871 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.599844 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c918c5c-994d-4531-ad18-89ee658e2c4e-openshift-service-ca" pod "0c918c5c-994d-4531-ad18-89ee658e2c4e" (UID: "0c918c5c-994d-4531-ad18-89ee658e2c4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:33.600169 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.600148 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c918c5c-994d-4531-ad18-89ee658e2c4e-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:33.602482 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.602459 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-maas-api-tls" pod "0c918c5c-994d-4531-ad18-89ee658e2c4e" (UID: "0c918c5c-994d-4531-ad18-89ee658e2c4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:38:33.602565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.602486 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-metrics-certs" pod "0c918c5c-994d-4531-ad18-89ee658e2c4e" (UID: "0c918c5c-994d-4531-ad18-89ee658e2c4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:38:33.602565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.602519 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c918c5c-994d-4531-ad18-89ee658e2c4e-kube-api-access-tt4cl" pod "0c918c5c-994d-4531-ad18-89ee658e2c4e" (UID: "0c918c5c-994d-4531-ad18-89ee658e2c4e"). InnerVolumeSpecName "kube-api-access-tt4cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:33.727652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.706301 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt4cl\" (UniqueName: \"kubernetes.io/projected/0c918c5c-994d-4531-ad18-89ee658e2c4e-kube-api-access-tt4cl\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:33.727652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.706339 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:33.727652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:33.706354 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c918c5c-994d-4531-ad18-89ee658e2c4e-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:34.008072 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:34.008037 6389 generic.go:356] "Generic (PLEG): container finished" podID="0c918c5c-994d-4531-ad18-89ee658e2c4e" containerID="82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a" exitCode=0 Nov 03 14:38:34.008269 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:34.008104 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" Nov 03 14:38:34.008269 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:34.008115 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" event={"ID":"0c918c5c-994d-4531-ad18-89ee658e2c4e","Type":"ContainerDied","Data":"82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a"} Nov 03 14:38:34.008269 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:34.008199 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26" event={"ID":"0c918c5c-994d-4531-ad18-89ee658e2c4e","Type":"ContainerDied","Data":"cf01a144bf0a1dbe7cbf48fd5ab3a1e620ae25c6ab0028c4c3d4f1586e5f1674"} Nov 03 14:38:34.008269 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:34.008230 6389 scope.go:122] "RemoveContainer" containerID="82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a" Nov 03 14:38:34.017920 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:34.017902 6389 scope.go:122] "RemoveContainer" containerID="82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a" Nov 03 14:38:34.018210 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:34.018188 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a\": container with ID starting with 82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a not found: ID does not exist" containerID="82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a" Nov 03 14:38:34.018293 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:34.018221 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a"} err="failed to get container status \"82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a\": rpc error: code = NotFound desc = could not find container \"82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a\": container with ID starting with 82f2f1f459f6348ea343c24067c0dd4ca77ebedf02cf411d1f146928e488869a not found: ID does not exist" Nov 03 14:38:34.072694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:34.072602 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26"] Nov 03 14:38:34.085049 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:34.085022 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2d0da2-6f8cd78864-lcq26"] Nov 03 14:38:34.618942 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:34.618900 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c918c5c-994d-4531-ad18-89ee658e2c4e" path="/var/lib/kubelet/pods/0c918c5c-994d-4531-ad18-89ee658e2c4e/volumes" Nov 03 14:38:34.619517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:34.619495 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" path="/var/lib/kubelet/pods/305d8254-7294-4a52-afa2-0887d742f8f9/volumes" Nov 03 14:38:35.152286 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:35.152256 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8"] Nov 03 14:38:35.156419 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:35.156385 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:36.907106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907078 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7"] Nov 03 14:38:36.907596 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907582 6389 cpu_manager.go:414] "containerMap: removing container" podUID="30003229-866c-4a72-be45-37c3134e3159" containerName="istio-proxy" Nov 03 14:38:36.907662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907597 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30003229-866c-4a72-be45-37c3134e3159" containerName="istio-proxy" Nov 03 14:38:36.907662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907631 6389 cpu_manager.go:414] "containerMap: removing container" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" containerName="istio-proxy" Nov 03 14:38:36.907662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907638 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" containerName="istio-proxy" Nov 03 14:38:36.907662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907646 6389 cpu_manager.go:414] "containerMap: removing container" podUID="5c43724b-ae54-408e-8e4d-b307cd947d9c" containerName="istio-proxy" Nov 03 14:38:36.907662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907652 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c43724b-ae54-408e-8e4d-b307cd947d9c" containerName="istio-proxy" Nov 03 14:38:36.907820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907679 6389 cpu_manager.go:414] "containerMap: removing container" podUID="0c918c5c-994d-4531-ad18-89ee658e2c4e" containerName="maas-api" Nov 03 14:38:36.907820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907684 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c918c5c-994d-4531-ad18-89ee658e2c4e" containerName="maas-api" Nov 03 14:38:36.907820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907689 6389 cpu_manager.go:414] "containerMap: removing container" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" Nov 03 14:38:36.907820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907694 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" Nov 03 14:38:36.907820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907766 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c43724b-ae54-408e-8e4d-b307cd947d9c" containerName="istio-proxy" Nov 03 14:38:36.907820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907774 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c918c5c-994d-4531-ad18-89ee658e2c4e" containerName="maas-api" Nov 03 14:38:36.907820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907782 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0cd10d-efc2-47f3-860c-5db6ed4b38db" containerName="istio-proxy" Nov 03 14:38:36.907820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907789 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="30003229-866c-4a72-be45-37c3134e3159" containerName="istio-proxy" Nov 03 14:38:36.907820 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.907796 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="305d8254-7294-4a52-afa2-0887d742f8f9" containerName="istio-proxy" Nov 03 14:38:36.912431 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.912411 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:36.935961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.935933 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7"] Nov 03 14:38:36.942542 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.942516 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-e2e--e9e6f325-dockercfg-6vl2b\"" Nov 03 14:38:36.944196 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:36.944179 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-b44368991f6c\"" Nov 03 14:38:37.092909 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.092865 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:37.093105 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.092959 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:37.093105 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.093000 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfgz\" (UniqueName: \"kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-kube-api-access-hgfgz\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:37.093105 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.093072 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:37.181880 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.181835 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7"] Nov 03 14:38:37.182256 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:37.182228 6389 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hgfgz plugins-config-volume provider-credentials routing-overlay-volume], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" podUID="826a8e61-ce02-4d8c-9fbd-f98fa2075f38" Nov 03 14:38:37.194446 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.194416 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:37.194662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.194491 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:37.194662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.194524 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfgz\" (UniqueName: \"kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-kube-api-access-hgfgz\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:37.194662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.194594 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:37.195373 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.195349 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:37.195474 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.195433 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:37.195542 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:37.195526 6389 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b44368991f6c: secret "agc-cred-b44368991f6c" not found Nov 03 14:38:37.195582 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:37.195545 6389 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7: secret "agc-cred-b44368991f6c" not found Nov 03 14:38:37.195660 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:37.195650 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-provider-credentials podName:826a8e61-ce02-4d8c-9fbd-f98fa2075f38 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:37.695630311 +0000 UTC m=+1593.254266777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" (UID: "826a8e61-ce02-4d8c-9fbd-f98fa2075f38") : secret "agc-cred-b44368991f6c" not found Nov 03 14:38:37.216600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.216565 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfgz\" (UniqueName: \"kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-kube-api-access-hgfgz\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:37.706162 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:37.706133 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:37.706357 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:37.706245 6389 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b44368991f6c: secret "agc-cred-b44368991f6c" not found Nov 03 14:38:37.706357 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:37.706262 6389 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7: secret "agc-cred-b44368991f6c" not found Nov 03 14:38:37.706357 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:37.706319 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-provider-credentials podName:826a8e61-ce02-4d8c-9fbd-f98fa2075f38 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:38.706302947 +0000 UTC m=+1594.264939409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" (UID: "826a8e61-ce02-4d8c-9fbd-f98fa2075f38") : secret "agc-cred-b44368991f6c" not found Nov 03 14:38:38.076773 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:38.076690 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:38.086904 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:38.086869 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:38.211782 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:38.211737 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-routing-overlay-volume\") pod \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " Nov 03 14:38:38.211931 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:38.211891 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-plugins-config-volume\") pod \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " Nov 03 14:38:38.212039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:38.212022 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-kube-api-access-hgfgz\" (UniqueName: \"kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-kube-api-access-hgfgz\") pod \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " Nov 03 14:38:38.212157 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:38.212138 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-routing-overlay-volume" pod "826a8e61-ce02-4d8c-9fbd-f98fa2075f38" (UID: "826a8e61-ce02-4d8c-9fbd-f98fa2075f38"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:38.212270 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:38.212247 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-plugins-config-volume" pod "826a8e61-ce02-4d8c-9fbd-f98fa2075f38" (UID: "826a8e61-ce02-4d8c-9fbd-f98fa2075f38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:38.212512 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:38.212493 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:38.212512 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:38.212508 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:38.214271 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:38.214253 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-kube-api-access-hgfgz" pod "826a8e61-ce02-4d8c-9fbd-f98fa2075f38" (UID: "826a8e61-ce02-4d8c-9fbd-f98fa2075f38"). InnerVolumeSpecName "kube-api-access-hgfgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:38.314197 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:38.314161 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgfgz\" (UniqueName: \"kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-kube-api-access-hgfgz\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:38.719038 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:38.719004 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7\" (UID: \"826a8e61-ce02-4d8c-9fbd-f98fa2075f38\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:38.719248 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:38.719183 6389 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b44368991f6c: secret "agc-cred-b44368991f6c" not found Nov 03 14:38:38.719248 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:38.719205 6389 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7: secret "agc-cred-b44368991f6c" not found Nov 03 14:38:38.719383 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:38.719270 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-provider-credentials podName:826a8e61-ce02-4d8c-9fbd-f98fa2075f38 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:40.719251366 +0000 UTC m=+1596.277887842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" (UID: "826a8e61-ce02-4d8c-9fbd-f98fa2075f38") : secret "agc-cred-b44368991f6c" not found Nov 03 14:38:39.082565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:39.082534 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7" Nov 03 14:38:39.163027 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:39.162994 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7"] Nov 03 14:38:39.166780 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:39.166750 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846f7xvp7"] Nov 03 14:38:39.327117 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:39.327065 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/826a8e61-ce02-4d8c-9fbd-f98fa2075f38-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:39.753103 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:39.753073 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9"] Nov 03 14:38:39.753517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:39.753488 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" containerID="cri-o://4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0" gracePeriod=30 Nov 03 14:38:39.759828 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:39.759800 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg"] Nov 03 14:38:39.760059 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:39.760030 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" podUID="103e79b4-ac7a-4f53-affe-8a06ef5a3d46" containerName="istio-proxy" containerID="cri-o://83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c" gracePeriod=30 Nov 03 14:38:39.775818 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:39.775782 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw"] Nov 03 14:38:39.775990 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:39.775959 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" podUID="5fedcbdd-19f2-4b86-a328-3431ee04dc75" containerName="istio-proxy" containerID="cri-o://1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020" gracePeriod=30 Nov 03 14:38:39.778160 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:39.778135 6389 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:38:39.778287 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:39.778178 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:38:40.155785 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.155676 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" podUID="7d86ec84-3f92-45a7-a553-4f20322ac5be" containerName="payload-processing" containerID="cri-o://8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9" gracePeriod=25 Nov 03 14:38:40.437383 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.437357 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:40.452483 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.452461 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-provider-credentials\") pod \"7d86ec84-3f92-45a7-a553-4f20322ac5be\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " Nov 03 14:38:40.452642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.452521 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-plugins-config-volume\") pod \"7d86ec84-3f92-45a7-a553-4f20322ac5be\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " Nov 03 14:38:40.452689 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.452681 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-kube-api-access-6dkzg\" (UniqueName: \"kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-kube-api-access-6dkzg\") pod \"7d86ec84-3f92-45a7-a553-4f20322ac5be\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " Nov 03 14:38:40.452727 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.452708 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-routing-overlay-volume\") pod \"7d86ec84-3f92-45a7-a553-4f20322ac5be\" (UID: \"7d86ec84-3f92-45a7-a553-4f20322ac5be\") " Nov 03 14:38:40.452913 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.452885 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-plugins-config-volume" pod "7d86ec84-3f92-45a7-a553-4f20322ac5be" (UID: "7d86ec84-3f92-45a7-a553-4f20322ac5be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:40.453179 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.453162 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:40.453353 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.453330 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-routing-overlay-volume" pod "7d86ec84-3f92-45a7-a553-4f20322ac5be" (UID: "7d86ec84-3f92-45a7-a553-4f20322ac5be"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:40.454857 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.454834 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-provider-credentials" pod "7d86ec84-3f92-45a7-a553-4f20322ac5be" (UID: "7d86ec84-3f92-45a7-a553-4f20322ac5be"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:40.455154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.455130 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-kube-api-access-6dkzg" pod "7d86ec84-3f92-45a7-a553-4f20322ac5be" (UID: "7d86ec84-3f92-45a7-a553-4f20322ac5be"). InnerVolumeSpecName "kube-api-access-6dkzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:40.554771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.554743 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:40.554771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.554770 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dkzg\" (UniqueName: \"kubernetes.io/projected/7d86ec84-3f92-45a7-a553-4f20322ac5be-kube-api-access-6dkzg\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:40.554771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.554779 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d86ec84-3f92-45a7-a553-4f20322ac5be-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:40.618586 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:40.618551 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826a8e61-ce02-4d8c-9fbd-f98fa2075f38" path="/var/lib/kubelet/pods/826a8e61-ce02-4d8c-9fbd-f98fa2075f38/volumes" Nov 03 14:38:41.095188 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:41.095146 6389 generic.go:356] "Generic (PLEG): container finished" podID="7d86ec84-3f92-45a7-a553-4f20322ac5be" containerID="8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9" exitCode=0 Nov 03 14:38:41.095400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:41.095263 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" Nov 03 14:38:41.095400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:41.095291 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" event={"ID":"7d86ec84-3f92-45a7-a553-4f20322ac5be","Type":"ContainerDied","Data":"8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9"} Nov 03 14:38:41.095400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:41.095347 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8" event={"ID":"7d86ec84-3f92-45a7-a553-4f20322ac5be","Type":"ContainerDied","Data":"16fdf9b514b40e889716bc2c1d61efa5d1013091ae365b3d148dd99c9183df3a"} Nov 03 14:38:41.095400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:41.095370 6389 scope.go:122] "RemoveContainer" containerID="8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9" Nov 03 14:38:41.107591 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:41.107540 6389 scope.go:122] "RemoveContainer" containerID="8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9" Nov 03 14:38:41.108911 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:41.108711 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9\": container with ID starting with 8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9 not found: ID does not exist" containerID="8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9" Nov 03 14:38:41.108911 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:41.108756 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9"} err="failed to get container status \"8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9\": rpc error: code = NotFound desc = could not find container \"8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9\": container with ID starting with 8033995d90d26db839dcc20909aef9a4be1cb07a742d342e726a14395ed1e3f9 not found: ID does not exist" Nov 03 14:38:41.153472 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:41.153443 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8"] Nov 03 14:38:41.165697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:41.165668 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d786b7s8"] Nov 03 14:38:42.617695 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:42.617665 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d86ec84-3f92-45a7-a553-4f20322ac5be" path="/var/lib/kubelet/pods/7d86ec84-3f92-45a7-a553-4f20322ac5be/volumes" Nov 03 14:38:45.045591 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.045518 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:38:45.078968 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.078925 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:38:45.106271 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.106246 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:38:45.107335 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107317 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-data\") pod \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " Nov 03 14:38:45.107416 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107355 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istiod-ca-cert\") pod \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " Nov 03 14:38:45.107416 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107385 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-podinfo\") pod \"4464194f-c9e3-4bfa-951c-1b8897ef028c\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " Nov 03 14:38:45.107532 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107418 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4464194f-c9e3-4bfa-951c-1b8897ef028c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4464194f-c9e3-4bfa-951c-1b8897ef028c-istiod-ca-cert\") pod \"4464194f-c9e3-4bfa-951c-1b8897ef028c\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " Nov 03 14:38:45.107588 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107547 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-socket\") pod \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " Nov 03 14:38:45.107733 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107600 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-kube-api-access-m9ks4\" (UniqueName: \"kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-kube-api-access-m9ks4\") pod \"4464194f-c9e3-4bfa-951c-1b8897ef028c\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " Nov 03 14:38:45.107733 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107645 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-token\" (UniqueName: \"kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-token\") pod \"4464194f-c9e3-4bfa-951c-1b8897ef028c\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " Nov 03 14:38:45.107733 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107682 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-data" pod "103e79b4-ac7a-4f53-affe-8a06ef5a3d46" (UID: "103e79b4-ac7a-4f53-affe-8a06ef5a3d46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.107733 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107694 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-kube-api-access-86h9w\" (UniqueName: \"kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-kube-api-access-86h9w\") pod \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " Nov 03 14:38:45.107936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107782 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-certs\") pod \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " Nov 03 14:38:45.107936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107804 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4464194f-c9e3-4bfa-951c-1b8897ef028c-istiod-ca-cert" pod "4464194f-c9e3-4bfa-951c-1b8897ef028c" (UID: "4464194f-c9e3-4bfa-951c-1b8897ef028c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:45.107936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107811 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istiod-ca-cert" pod "103e79b4-ac7a-4f53-affe-8a06ef5a3d46" (UID: "103e79b4-ac7a-4f53-affe-8a06ef5a3d46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:45.107936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107827 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-envoy\") pod \"4464194f-c9e3-4bfa-951c-1b8897ef028c\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " Nov 03 14:38:45.107936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107874 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-credential-socket\") pod \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " Nov 03 14:38:45.107936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107897 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-token\" (UniqueName: \"kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-token\") pod \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " Nov 03 14:38:45.107936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107932 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-socket\") pod \"4464194f-c9e3-4bfa-951c-1b8897ef028c\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " Nov 03 14:38:45.108292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.107975 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-data\") pod \"4464194f-c9e3-4bfa-951c-1b8897ef028c\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " Nov 03 14:38:45.108292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108015 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-envoy\") pod \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " Nov 03 14:38:45.108292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108038 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-certs\") pod \"4464194f-c9e3-4bfa-951c-1b8897ef028c\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " Nov 03 14:38:45.108292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108072 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-credential-socket\") pod \"4464194f-c9e3-4bfa-951c-1b8897ef028c\" (UID: \"4464194f-c9e3-4bfa-951c-1b8897ef028c\") " Nov 03 14:38:45.108292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108083 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-certs" pod "103e79b4-ac7a-4f53-affe-8a06ef5a3d46" (UID: "103e79b4-ac7a-4f53-affe-8a06ef5a3d46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.108292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108096 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-podinfo\") pod \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\" (UID: \"103e79b4-ac7a-4f53-affe-8a06ef5a3d46\") " Nov 03 14:38:45.108792 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108436 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-socket" pod "103e79b4-ac7a-4f53-affe-8a06ef5a3d46" (UID: "103e79b4-ac7a-4f53-affe-8a06ef5a3d46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.108792 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108454 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.108792 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108469 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4464194f-c9e3-4bfa-951c-1b8897ef028c-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.108792 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108481 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.108792 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108493 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.108792 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108763 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-certs" pod "4464194f-c9e3-4bfa-951c-1b8897ef028c" (UID: "4464194f-c9e3-4bfa-951c-1b8897ef028c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.109122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108881 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-credential-socket" pod "4464194f-c9e3-4bfa-951c-1b8897ef028c" (UID: "4464194f-c9e3-4bfa-951c-1b8897ef028c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.109122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.108992 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-data" pod "4464194f-c9e3-4bfa-951c-1b8897ef028c" (UID: "4464194f-c9e3-4bfa-951c-1b8897ef028c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.109122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.109107 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-credential-socket" pod "103e79b4-ac7a-4f53-affe-8a06ef5a3d46" (UID: "103e79b4-ac7a-4f53-affe-8a06ef5a3d46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.109283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.109164 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-socket" pod "4464194f-c9e3-4bfa-951c-1b8897ef028c" (UID: "4464194f-c9e3-4bfa-951c-1b8897ef028c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.110258 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.110237 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-envoy" pod "4464194f-c9e3-4bfa-951c-1b8897ef028c" (UID: "4464194f-c9e3-4bfa-951c-1b8897ef028c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.110477 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.110461 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-kube-api-access-m9ks4" pod "4464194f-c9e3-4bfa-951c-1b8897ef028c" (UID: "4464194f-c9e3-4bfa-951c-1b8897ef028c"). InnerVolumeSpecName "kube-api-access-m9ks4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.110689 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.110663 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-kube-api-access-86h9w" pod "103e79b4-ac7a-4f53-affe-8a06ef5a3d46" (UID: "103e79b4-ac7a-4f53-affe-8a06ef5a3d46"). InnerVolumeSpecName "kube-api-access-86h9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.110980 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.110954 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-envoy" pod "103e79b4-ac7a-4f53-affe-8a06ef5a3d46" (UID: "103e79b4-ac7a-4f53-affe-8a06ef5a3d46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.111079 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.111056 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-token" pod "103e79b4-ac7a-4f53-affe-8a06ef5a3d46" (UID: "103e79b4-ac7a-4f53-affe-8a06ef5a3d46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.111313 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.111289 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-podinfo" pod "4464194f-c9e3-4bfa-951c-1b8897ef028c" (UID: "4464194f-c9e3-4bfa-951c-1b8897ef028c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:45.111378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.111336 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-podinfo" pod "103e79b4-ac7a-4f53-affe-8a06ef5a3d46" (UID: "103e79b4-ac7a-4f53-affe-8a06ef5a3d46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:45.112239 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.112220 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-token" pod "4464194f-c9e3-4bfa-951c-1b8897ef028c" (UID: "4464194f-c9e3-4bfa-951c-1b8897ef028c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.120432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.120407 6389 generic.go:356] "Generic (PLEG): container finished" podID="103e79b4-ac7a-4f53-affe-8a06ef5a3d46" containerID="83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c" exitCode=0 Nov 03 14:38:45.120511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.120449 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" Nov 03 14:38:45.120511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.120492 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" event={"ID":"103e79b4-ac7a-4f53-affe-8a06ef5a3d46","Type":"ContainerDied","Data":"83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c"} Nov 03 14:38:45.120631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.120543 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg" event={"ID":"103e79b4-ac7a-4f53-affe-8a06ef5a3d46","Type":"ContainerDied","Data":"9ef45ad5203cc3202775a79fc809293fb827f84c1fefe48f95c7268060968f4c"} Nov 03 14:38:45.120631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.120572 6389 scope.go:122] "RemoveContainer" containerID="83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c" Nov 03 14:38:45.122284 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.122240 6389 generic.go:356] "Generic (PLEG): container finished" podID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerID="4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0" exitCode=0 Nov 03 14:38:45.122413 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.122397 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" Nov 03 14:38:45.125678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.122395 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" event={"ID":"4464194f-c9e3-4bfa-951c-1b8897ef028c","Type":"ContainerDied","Data":"4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0"} Nov 03 14:38:45.125822 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.125716 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9" event={"ID":"4464194f-c9e3-4bfa-951c-1b8897ef028c","Type":"ContainerDied","Data":"b0576dae4e8a4062dff4ca6dfa8498c446ff72181e27e4700394a35f8a8e3525"} Nov 03 14:38:45.126677 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.126655 6389 generic.go:356] "Generic (PLEG): container finished" podID="5fedcbdd-19f2-4b86-a328-3431ee04dc75" containerID="1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020" exitCode=0 Nov 03 14:38:45.126764 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.126736 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" event={"ID":"5fedcbdd-19f2-4b86-a328-3431ee04dc75","Type":"ContainerDied","Data":"1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020"} Nov 03 14:38:45.126764 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.126763 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" event={"ID":"5fedcbdd-19f2-4b86-a328-3431ee04dc75","Type":"ContainerDied","Data":"468dea942e03dade6a1c744bf6b9fd7eb83ebd7713034b1783733f5fb1ce294b"} Nov 03 14:38:45.126889 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.126817 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw" Nov 03 14:38:45.138778 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.137088 6389 scope.go:122] "RemoveContainer" containerID="83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c" Nov 03 14:38:45.138778 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:45.137464 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c\": container with ID starting with 83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c not found: ID does not exist" containerID="83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c" Nov 03 14:38:45.138778 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.137499 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c"} err="failed to get container status \"83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c\": rpc error: code = NotFound desc = could not find container \"83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c\": container with ID starting with 83b0951aa90b95f4d75bce9ec5e60120dc55d280119402bdc861f9db4277c27c not found: ID does not exist" Nov 03 14:38:45.138778 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.137521 6389 scope.go:122] "RemoveContainer" containerID="4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0" Nov 03 14:38:45.146810 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.146791 6389 scope.go:122] "RemoveContainer" containerID="4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0" Nov 03 14:38:45.147078 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:45.147060 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0\": container with ID starting with 4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0 not found: ID does not exist" containerID="4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0" Nov 03 14:38:45.147141 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.147086 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0"} err="failed to get container status \"4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0\": rpc error: code = NotFound desc = could not find container \"4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0\": container with ID starting with 4445519df9aee495b91a0a52040a9daf261419243a776b8b511cf681177a67a0 not found: ID does not exist" Nov 03 14:38:45.147141 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.147103 6389 scope.go:122] "RemoveContainer" containerID="1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020" Nov 03 14:38:45.156123 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.156092 6389 scope.go:122] "RemoveContainer" containerID="1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020" Nov 03 14:38:45.156406 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:45.156385 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020\": container with ID starting with 1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020 not found: ID does not exist" containerID="1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020" Nov 03 14:38:45.156468 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.156411 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020"} err="failed to get container status \"1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020\": rpc error: code = NotFound desc = could not find container \"1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020\": container with ID starting with 1667a12a4117524f8f4e8e736b3141488aa0f56c76add7685072919b5351a020 not found: ID does not exist" Nov 03 14:38:45.208855 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.208828 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istiod-ca-cert\") pod \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " Nov 03 14:38:45.209060 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.208884 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-socket\") pod \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " Nov 03 14:38:45.209060 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.208922 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-kube-api-access-dmqw8\" (UniqueName: \"kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-kube-api-access-dmqw8\") pod \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " Nov 03 14:38:45.209060 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.208952 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-envoy\") pod \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " Nov 03 14:38:45.209244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.209110 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-podinfo\") pod \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " Nov 03 14:38:45.209244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.209149 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-certs\") pod \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " Nov 03 14:38:45.209244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.209200 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-token\" (UniqueName: \"kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-token\") pod \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " Nov 03 14:38:45.209409 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.209247 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-credential-socket\") pod \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " Nov 03 14:38:45.209409 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.209258 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-socket" pod "5fedcbdd-19f2-4b86-a328-3431ee04dc75" (UID: "5fedcbdd-19f2-4b86-a328-3431ee04dc75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.209409 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.209280 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-data\") pod \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\" (UID: \"5fedcbdd-19f2-4b86-a328-3431ee04dc75\") " Nov 03 14:38:45.209409 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.209301 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istiod-ca-cert" pod "5fedcbdd-19f2-4b86-a328-3431ee04dc75" (UID: "5fedcbdd-19f2-4b86-a328-3431ee04dc75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:45.209626 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.209476 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-certs" pod "5fedcbdd-19f2-4b86-a328-3431ee04dc75" (UID: "5fedcbdd-19f2-4b86-a328-3431ee04dc75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.209740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.209716 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-credential-socket" pod "5fedcbdd-19f2-4b86-a328-3431ee04dc75" (UID: "5fedcbdd-19f2-4b86-a328-3431ee04dc75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.209740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.209731 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-data" pod "5fedcbdd-19f2-4b86-a328-3431ee04dc75" (UID: "5fedcbdd-19f2-4b86-a328-3431ee04dc75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210057 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210083 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210099 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210114 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210128 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9ks4\" (UniqueName: \"kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-kube-api-access-m9ks4\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210143 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210157 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86h9w\" (UniqueName: \"kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-kube-api-access-86h9w\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210172 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210186 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210199 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210212 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210226 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210239 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210253 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210267 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210281 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210293 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103e79b4-ac7a-4f53-affe-8a06ef5a3d46-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210305 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4464194f-c9e3-4bfa-951c-1b8897ef028c-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.210642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.210319 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.211979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.211950 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-podinfo" pod "5fedcbdd-19f2-4b86-a328-3431ee04dc75" (UID: "5fedcbdd-19f2-4b86-a328-3431ee04dc75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:45.211979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.211955 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-token" pod "5fedcbdd-19f2-4b86-a328-3431ee04dc75" (UID: "5fedcbdd-19f2-4b86-a328-3431ee04dc75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.212102 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.211985 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-envoy" pod "5fedcbdd-19f2-4b86-a328-3431ee04dc75" (UID: "5fedcbdd-19f2-4b86-a328-3431ee04dc75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.212102 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.212014 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-kube-api-access-dmqw8" pod "5fedcbdd-19f2-4b86-a328-3431ee04dc75" (UID: "5fedcbdd-19f2-4b86-a328-3431ee04dc75"). InnerVolumeSpecName "kube-api-access-dmqw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.225502 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.225476 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg"] Nov 03 14:38:45.229566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.229546 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-wmmsg"] Nov 03 14:38:45.311234 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.311144 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.311234 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.311183 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.311234 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.311201 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmqw8\" (UniqueName: \"kubernetes.io/projected/5fedcbdd-19f2-4b86-a328-3431ee04dc75-kube-api-access-dmqw8\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.311234 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.311230 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fedcbdd-19f2-4b86-a328-3431ee04dc75-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.314441 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.314419 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9"] Nov 03 14:38:45.320871 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.320847 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-t5jm9"] Nov 03 14:38:45.476196 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.476166 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw"] Nov 03 14:38:45.479559 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:45.479532 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-29srw"] Nov 03 14:38:46.618897 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:46.618868 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103e79b4-ac7a-4f53-affe-8a06ef5a3d46" path="/var/lib/kubelet/pods/103e79b4-ac7a-4f53-affe-8a06ef5a3d46/volumes" Nov 03 14:38:46.619355 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:46.619341 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" path="/var/lib/kubelet/pods/4464194f-c9e3-4bfa-951c-1b8897ef028c/volumes" Nov 03 14:38:46.619770 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:38:46.619758 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fedcbdd-19f2-4b86-a328-3431ee04dc75" path="/var/lib/kubelet/pods/5fedcbdd-19f2-4b86-a328-3431ee04dc75/volumes" Nov 03 14:38:58.382331 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:38:58.382283 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:39:04.265822 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:04.265793 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5"] Nov 03 14:39:04.266247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:04.266017 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" podUID="b8858f0b-63bc-4fc7-9626-fac6e74a9352" containerName="istio-proxy" containerID="cri-o://5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0" gracePeriod=30 Nov 03 14:39:04.277913 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:04.277883 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52"] Nov 03 14:39:04.278119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:04.278097 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" podUID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" containerName="istio-proxy" containerID="cri-o://c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b" gracePeriod=30 Nov 03 14:39:09.560630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.560580 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:39:09.567765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.567738 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istiod-ca-cert\") pod \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " Nov 03 14:39:09.567897 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.567816 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-kube-api-access-r584j\" (UniqueName: \"kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-kube-api-access-r584j\") pod \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " Nov 03 14:39:09.567897 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.567850 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-token\" (UniqueName: \"kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-token\") pod \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " Nov 03 14:39:09.567897 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.567881 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-credential-socket\") pod \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " Nov 03 14:39:09.568071 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.567919 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-data\") pod \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " Nov 03 14:39:09.568071 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.568013 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-socket\") pod \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " Nov 03 14:39:09.568071 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.568046 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istiod-ca-cert" pod "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" (UID: "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:09.568244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.568090 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-certs\") pod \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " Nov 03 14:39:09.568244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.568119 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-podinfo\") pod \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " Nov 03 14:39:09.568244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.568179 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-envoy\") pod \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\" (UID: \"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1\") " Nov 03 14:39:09.568571 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.568553 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.568908 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.568883 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-credential-socket" pod "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" (UID: "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.568908 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.568893 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-data" pod "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" (UID: "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.569080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.568994 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-socket" pod "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" (UID: "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.569340 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.569266 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-certs" pod "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" (UID: "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.570960 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.570937 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-kube-api-access-r584j" pod "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" (UID: "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1"). InnerVolumeSpecName "kube-api-access-r584j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:09.571085 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.571067 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-envoy" pod "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" (UID: "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.571181 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.571114 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-token" pod "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" (UID: "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:09.571375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.571356 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-podinfo" pod "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" (UID: "5be2ce35-3ee0-4aa8-88c9-19d23c597ca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:39:09.585878 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.585846 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:39:09.669930 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.669276 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-envoy\") pod \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " Nov 03 14:39:09.669930 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.669320 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-podinfo\") pod \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " Nov 03 14:39:09.669930 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.669415 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-credential-socket\") pod \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " Nov 03 14:39:09.669930 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.669455 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-token\" (UniqueName: \"kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-token\") pod \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " Nov 03 14:39:09.669930 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.669487 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-socket\") pod \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " Nov 03 14:39:09.669930 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.669549 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-data\") pod \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " Nov 03 14:39:09.669930 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.669623 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-certs\") pod \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " Nov 03 14:39:09.669930 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.669676 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istiod-ca-cert\") pod \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " Nov 03 14:39:09.669930 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.669753 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-kube-api-access-wkvb5\" (UniqueName: \"kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-kube-api-access-wkvb5\") pod \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\" (UID: \"b8858f0b-63bc-4fc7-9626-fac6e74a9352\") " Nov 03 14:39:09.670706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.670680 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-certs" pod "b8858f0b-63bc-4fc7-9626-fac6e74a9352" (UID: "b8858f0b-63bc-4fc7-9626-fac6e74a9352"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.670706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.670721 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-data" pod "b8858f0b-63bc-4fc7-9626-fac6e74a9352" (UID: "b8858f0b-63bc-4fc7-9626-fac6e74a9352"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.670706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.670975 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istiod-ca-cert" pod "b8858f0b-63bc-4fc7-9626-fac6e74a9352" (UID: "b8858f0b-63bc-4fc7-9626-fac6e74a9352"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:09.670706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.670947 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-socket" pod "b8858f0b-63bc-4fc7-9626-fac6e74a9352" (UID: "b8858f0b-63bc-4fc7-9626-fac6e74a9352"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.671514 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.671238 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-credential-socket" pod "b8858f0b-63bc-4fc7-9626-fac6e74a9352" (UID: "b8858f0b-63bc-4fc7-9626-fac6e74a9352"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.672017 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.671985 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.672017 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.672013 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.672180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.672027 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.672180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.672039 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.672180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.672049 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.672180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.672057 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.672180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.672065 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.672180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.672073 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.672180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.672081 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r584j\" (UniqueName: \"kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-kube-api-access-r584j\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.672180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.672089 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.672180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.672097 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.672180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.672105 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.672180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.672112 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.675183 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.673247 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-envoy" pod "b8858f0b-63bc-4fc7-9626-fac6e74a9352" (UID: "b8858f0b-63bc-4fc7-9626-fac6e74a9352"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.675183 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.673722 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-kube-api-access-wkvb5" pod "b8858f0b-63bc-4fc7-9626-fac6e74a9352" (UID: "b8858f0b-63bc-4fc7-9626-fac6e74a9352"). InnerVolumeSpecName "kube-api-access-wkvb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:09.675183 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.674794 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-token" pod "b8858f0b-63bc-4fc7-9626-fac6e74a9352" (UID: "b8858f0b-63bc-4fc7-9626-fac6e74a9352"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:09.675183 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.674889 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-podinfo" pod "b8858f0b-63bc-4fc7-9626-fac6e74a9352" (UID: "b8858f0b-63bc-4fc7-9626-fac6e74a9352"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:39:09.773132 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.773103 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkvb5\" (UniqueName: \"kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-kube-api-access-wkvb5\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.773132 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.773129 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.773132 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.773138 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.773406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:09.773149 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8858f0b-63bc-4fc7-9626-fac6e74a9352-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:10.300249 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.300218 6389 generic.go:356] "Generic (PLEG): container finished" podID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" containerID="c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b" exitCode=0 Nov 03 14:39:10.300394 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.300275 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" Nov 03 14:39:10.300394 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.300308 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" event={"ID":"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1","Type":"ContainerDied","Data":"c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b"} Nov 03 14:39:10.300394 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.300360 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52" event={"ID":"5be2ce35-3ee0-4aa8-88c9-19d23c597ca1","Type":"ContainerDied","Data":"6300aef84c7c97f8d5bac067b3d0ab162b55563929de94a8b3dd38ce0a47cca7"} Nov 03 14:39:10.300394 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.300390 6389 scope.go:122] "RemoveContainer" containerID="c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b" Nov 03 14:39:10.301807 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.301784 6389 generic.go:356] "Generic (PLEG): container finished" podID="b8858f0b-63bc-4fc7-9626-fac6e74a9352" containerID="5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0" exitCode=0 Nov 03 14:39:10.301981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.301840 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" Nov 03 14:39:10.301981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.301891 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" event={"ID":"b8858f0b-63bc-4fc7-9626-fac6e74a9352","Type":"ContainerDied","Data":"5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0"} Nov 03 14:39:10.301981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.301944 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5" event={"ID":"b8858f0b-63bc-4fc7-9626-fac6e74a9352","Type":"ContainerDied","Data":"6155ef2b0c948fda3eb70400a81c110093f37b97f07c536bac5881c1460388eb"} Nov 03 14:39:10.311180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.311156 6389 scope.go:122] "RemoveContainer" containerID="c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b" Nov 03 14:39:10.311556 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:39:10.311529 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b\": container with ID starting with c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b not found: ID does not exist" containerID="c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b" Nov 03 14:39:10.311701 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.311566 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b"} err="failed to get container status \"c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b\": rpc error: code = NotFound desc = could not find container \"c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b\": container with ID starting with c627bb94524439c517d038a97c968592ab1799f8317685458f4365a14ea2697b not found: ID does not exist" Nov 03 14:39:10.311701 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.311588 6389 scope.go:122] "RemoveContainer" containerID="5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0" Nov 03 14:39:10.320795 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.320778 6389 scope.go:122] "RemoveContainer" containerID="5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0" Nov 03 14:39:10.321099 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:39:10.321073 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0\": container with ID starting with 5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0 not found: ID does not exist" containerID="5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0" Nov 03 14:39:10.321202 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.321099 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0"} err="failed to get container status \"5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0\": rpc error: code = NotFound desc = could not find container \"5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0\": container with ID starting with 5b09b68eed7d7e1a2809f8067d50cce272606b0f9d9c8b18cb4f5483c88702c0 not found: ID does not exist" Nov 03 14:39:10.372479 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.372450 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52"] Nov 03 14:39:10.385021 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.384996 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-7ds52"] Nov 03 14:39:10.414921 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.414897 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5"] Nov 03 14:39:10.420118 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.420094 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-ss8x5"] Nov 03 14:39:10.619759 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.619663 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" path="/var/lib/kubelet/pods/5be2ce35-3ee0-4aa8-88c9-19d23c597ca1/volumes" Nov 03 14:39:10.620146 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:10.620132 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8858f0b-63bc-4fc7-9626-fac6e74a9352" path="/var/lib/kubelet/pods/b8858f0b-63bc-4fc7-9626-fac6e74a9352/volumes" Nov 03 14:39:13.449083 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:13.448993 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb"] Nov 03 14:39:13.449563 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:13.449252 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" podUID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" containerName="istio-proxy" containerID="cri-o://6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8" gracePeriod=30 Nov 03 14:39:18.125637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:18.123733 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:39:18.125637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:18.123820 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:39:18.645475 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:18.645435 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 14:39:18.665180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:18.665122 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" podUID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 14:39:19.041905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.041782 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:39:19.136059 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.135971 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-kube-api-access-lqlrw\" (UniqueName: \"kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-kube-api-access-lqlrw\") pod \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " Nov 03 14:39:19.136059 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.136032 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-socket\") pod \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " Nov 03 14:39:19.136583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.136099 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istiod-ca-cert\") pod \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " Nov 03 14:39:19.136583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.136140 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-token\" (UniqueName: \"kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-token\") pod \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " Nov 03 14:39:19.136583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.136173 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-envoy\") pod \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " Nov 03 14:39:19.136583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.136227 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-certs\") pod \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " Nov 03 14:39:19.136583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.136299 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-credential-socket\") pod \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " Nov 03 14:39:19.136583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.136333 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-podinfo\") pod \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " Nov 03 14:39:19.136583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.136390 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-data\") pod \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\" (UID: \"4a69b4c1-74d0-4515-b0a0-1f7614b0a138\") " Nov 03 14:39:19.137998 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.137320 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-socket" pod "4a69b4c1-74d0-4515-b0a0-1f7614b0a138" (UID: "4a69b4c1-74d0-4515-b0a0-1f7614b0a138"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:19.137998 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.137352 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-certs" pod "4a69b4c1-74d0-4515-b0a0-1f7614b0a138" (UID: "4a69b4c1-74d0-4515-b0a0-1f7614b0a138"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:19.137998 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.137513 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-data" pod "4a69b4c1-74d0-4515-b0a0-1f7614b0a138" (UID: "4a69b4c1-74d0-4515-b0a0-1f7614b0a138"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:19.137998 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.137713 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-credential-socket" pod "4a69b4c1-74d0-4515-b0a0-1f7614b0a138" (UID: "4a69b4c1-74d0-4515-b0a0-1f7614b0a138"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:19.137998 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.137892 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istiod-ca-cert" pod "4a69b4c1-74d0-4515-b0a0-1f7614b0a138" (UID: "4a69b4c1-74d0-4515-b0a0-1f7614b0a138"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:19.142301 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.142264 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-kube-api-access-lqlrw" pod "4a69b4c1-74d0-4515-b0a0-1f7614b0a138" (UID: "4a69b4c1-74d0-4515-b0a0-1f7614b0a138"). InnerVolumeSpecName "kube-api-access-lqlrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:19.142535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.142498 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-podinfo" pod "4a69b4c1-74d0-4515-b0a0-1f7614b0a138" (UID: "4a69b4c1-74d0-4515-b0a0-1f7614b0a138"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:39:19.142962 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.142926 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-token" pod "4a69b4c1-74d0-4515-b0a0-1f7614b0a138" (UID: "4a69b4c1-74d0-4515-b0a0-1f7614b0a138"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:19.144693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.144668 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-envoy" pod "4a69b4c1-74d0-4515-b0a0-1f7614b0a138" (UID: "4a69b4c1-74d0-4515-b0a0-1f7614b0a138"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:19.238163 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.238128 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:19.238163 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.238157 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:19.238163 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.238167 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:19.238397 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.238176 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:19.238397 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.238186 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqlrw\" (UniqueName: \"kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-kube-api-access-lqlrw\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:19.238397 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.238196 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:19.238397 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.238205 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:19.238397 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.238214 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:19.238397 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.238221 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a69b4c1-74d0-4515-b0a0-1f7614b0a138-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:19.389291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.389205 6389 generic.go:356] "Generic (PLEG): container finished" podID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" containerID="6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8" exitCode=0 Nov 03 14:39:19.389291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.389253 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" Nov 03 14:39:19.389488 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.389298 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" event={"ID":"4a69b4c1-74d0-4515-b0a0-1f7614b0a138","Type":"ContainerDied","Data":"6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8"} Nov 03 14:39:19.389488 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.389354 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb" event={"ID":"4a69b4c1-74d0-4515-b0a0-1f7614b0a138","Type":"ContainerDied","Data":"768379f1f4aa4ba1f663490214298bd2e924b42ce202dec3861e33e6b07842e2"} Nov 03 14:39:19.389488 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.389377 6389 scope.go:122] "RemoveContainer" containerID="6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8" Nov 03 14:39:19.399095 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.399077 6389 scope.go:122] "RemoveContainer" containerID="6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8" Nov 03 14:39:19.399368 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:39:19.399348 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8\": container with ID starting with 6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8 not found: ID does not exist" containerID="6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8" Nov 03 14:39:19.399436 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.399376 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8"} err="failed to get container status \"6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8\": rpc error: code = NotFound desc = could not find container \"6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8\": container with ID starting with 6c3b98f02bea4242f303e35a4ed56492e5d54756ff524b182e130a01d4e40df8 not found: ID does not exist" Nov 03 14:39:19.445098 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.445067 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb"] Nov 03 14:39:19.457645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:19.457616 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-tgkbb"] Nov 03 14:39:20.616689 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:20.616656 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" path="/var/lib/kubelet/pods/4a69b4c1-74d0-4515-b0a0-1f7614b0a138/volumes" Nov 03 14:39:28.374469 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:39:28.374428 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:39:39.614698 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.614664 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4"] Nov 03 14:39:39.615113 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615091 6389 cpu_manager.go:414] "containerMap: removing container" podUID="b8858f0b-63bc-4fc7-9626-fac6e74a9352" containerName="istio-proxy" Nov 03 14:39:39.615113 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615101 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8858f0b-63bc-4fc7-9626-fac6e74a9352" containerName="istio-proxy" Nov 03 14:39:39.615199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615120 6389 cpu_manager.go:414] "containerMap: removing container" podUID="7d86ec84-3f92-45a7-a553-4f20322ac5be" containerName="payload-processing" Nov 03 14:39:39.615199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615125 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d86ec84-3f92-45a7-a553-4f20322ac5be" containerName="payload-processing" Nov 03 14:39:39.615199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615133 6389 cpu_manager.go:414] "containerMap: removing container" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" Nov 03 14:39:39.615199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615137 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" Nov 03 14:39:39.615199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615154 6389 cpu_manager.go:414] "containerMap: removing container" podUID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" containerName="istio-proxy" Nov 03 14:39:39.615199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615159 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" containerName="istio-proxy" Nov 03 14:39:39.615199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615165 6389 cpu_manager.go:414] "containerMap: removing container" podUID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" containerName="istio-proxy" Nov 03 14:39:39.615199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615177 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" containerName="istio-proxy" Nov 03 14:39:39.615199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615185 6389 cpu_manager.go:414] "containerMap: removing container" podUID="5fedcbdd-19f2-4b86-a328-3431ee04dc75" containerName="istio-proxy" Nov 03 14:39:39.615199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615190 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fedcbdd-19f2-4b86-a328-3431ee04dc75" containerName="istio-proxy" Nov 03 14:39:39.615199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615202 6389 cpu_manager.go:414] "containerMap: removing container" podUID="103e79b4-ac7a-4f53-affe-8a06ef5a3d46" containerName="istio-proxy" Nov 03 14:39:39.615516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615206 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103e79b4-ac7a-4f53-affe-8a06ef5a3d46" containerName="istio-proxy" Nov 03 14:39:39.615516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615274 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="103e79b4-ac7a-4f53-affe-8a06ef5a3d46" containerName="istio-proxy" Nov 03 14:39:39.615516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615282 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d86ec84-3f92-45a7-a553-4f20322ac5be" containerName="payload-processing" Nov 03 14:39:39.615516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615290 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="4464194f-c9e3-4bfa-951c-1b8897ef028c" containerName="istio-proxy" Nov 03 14:39:39.615516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615299 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a69b4c1-74d0-4515-b0a0-1f7614b0a138" containerName="istio-proxy" Nov 03 14:39:39.615516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615305 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8858f0b-63bc-4fc7-9626-fac6e74a9352" containerName="istio-proxy" Nov 03 14:39:39.615516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615310 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be2ce35-3ee0-4aa8-88c9-19d23c597ca1" containerName="istio-proxy" Nov 03 14:39:39.615516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.615315 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fedcbdd-19f2-4b86-a328-3431ee04dc75" containerName="istio-proxy" Nov 03 14:39:39.618578 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.618559 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.651346 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.651311 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4"] Nov 03 14:39:39.651927 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.651884 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9c7ea2cd2091\"" Nov 03 14:39:39.652118 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.652011 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mvlss\"" Nov 03 14:39:39.652118 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.652041 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 14:39:39.652118 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.652105 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 14:39:39.652118 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.652115 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 14:39:39.652381 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.652179 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 14:39:39.714244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.714205 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxfnw\" (UniqueName: \"kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-kube-api-access-gxfnw\") pod \"payload-processing-external-model-75cb49bc4f-gj7p4\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.715157 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.715132 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-routing-overlay-volume\") pod \"payload-processing-external-model-75cb49bc4f-gj7p4\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.716972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.716940 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-plugins-config-volume\") pod \"payload-processing-external-model-75cb49bc4f-gj7p4\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.717262 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.717246 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-provider-credentials\") pod \"payload-processing-external-model-75cb49bc4f-gj7p4\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.777953 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.777912 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz"] Nov 03 14:39:39.818975 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.818947 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-plugins-config-volume\") pod \"payload-processing-external-model-75cb49bc4f-gj7p4\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.819153 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.818983 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-provider-credentials\") pod \"payload-processing-external-model-75cb49bc4f-gj7p4\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.819153 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.819013 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxfnw\" (UniqueName: \"kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-kube-api-access-gxfnw\") pod \"payload-processing-external-model-75cb49bc4f-gj7p4\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.819153 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.819114 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-routing-overlay-volume\") pod \"payload-processing-external-model-75cb49bc4f-gj7p4\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.819959 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.819784 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-routing-overlay-volume\") pod \"payload-processing-external-model-75cb49bc4f-gj7p4\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.820579 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.820391 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-plugins-config-volume\") pod \"payload-processing-external-model-75cb49bc4f-gj7p4\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.822856 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.822831 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-provider-credentials\") pod \"payload-processing-external-model-75cb49bc4f-gj7p4\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.845935 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.845908 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxfnw\" (UniqueName: \"kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-kube-api-access-gxfnw\") pod \"payload-processing-external-model-75cb49bc4f-gj7p4\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:39.933940 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:39.933904 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:40.266158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:40.266133 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4"] Nov 03 14:39:40.267819 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:39:40.267789 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142d92c8_a7e3_43fc_92bd_a8da55634087.slice/crio-061dfffca22c68507fc686d477b23cb0cd65ad633f7024e6cea87441f0aaa18a WatchSource:0}: Error finding container 061dfffca22c68507fc686d477b23cb0cd65ad633f7024e6cea87441f0aaa18a: Status 404 returned error can't find the container with id 061dfffca22c68507fc686d477b23cb0cd65ad633f7024e6cea87441f0aaa18a Nov 03 14:39:40.502946 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:40.502907 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" event={"ID":"142d92c8-a7e3-43fc-92bd-a8da55634087","Type":"ContainerStarted","Data":"9ce4f40d09adb2f4d5fa6520f7d1bb2f809ba74501b84e0da1ca4d778b361cb9"} Nov 03 14:39:40.503140 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:40.502988 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" event={"ID":"142d92c8-a7e3-43fc-92bd-a8da55634087","Type":"ContainerStarted","Data":"061dfffca22c68507fc686d477b23cb0cd65ad633f7024e6cea87441f0aaa18a"} Nov 03 14:39:40.503140 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:40.503048 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:40.545725 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:40.545582 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" podStartSLOduration=1.5455689700000002 podStartE2EDuration="1.54556897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:39:40.545035691 +0000 UTC m=+1656.103672187" watchObservedRunningTime="2026-11-03 14:39:40.54556897 +0000 UTC m=+1656.104205451" Nov 03 14:39:44.463459 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.463423 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd"] Nov 03 14:39:44.464013 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.463687 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" podUID="c58d70cb-6a02-44ac-b026-da52f2b1ad12" containerName="maas-api" containerID="cri-o://14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7" gracePeriod=30 Nov 03 14:39:44.767559 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.767528 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:39:44.779187 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.779139 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" podUID="1739d708-b46f-4045-862e-967f64afd54b" containerName="payload-pre-processing" containerID="cri-o://b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b" gracePeriod=25 Nov 03 14:39:44.786525 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.786500 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4"] Nov 03 14:39:44.802929 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.802896 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58d70cb-6a02-44ac-b026-da52f2b1ad12-kube-api-access-42cv9\" (UniqueName: \"kubernetes.io/projected/c58d70cb-6a02-44ac-b026-da52f2b1ad12-kube-api-access-42cv9\") pod \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " Nov 03 14:39:44.803103 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.802971 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-metrics-certs\") pod \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " Nov 03 14:39:44.803182 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.803146 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-maas-api-tls\") pod \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " Nov 03 14:39:44.803244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.803211 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58d70cb-6a02-44ac-b026-da52f2b1ad12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58d70cb-6a02-44ac-b026-da52f2b1ad12-openshift-service-ca\") pod \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\" (UID: \"c58d70cb-6a02-44ac-b026-da52f2b1ad12\") " Nov 03 14:39:44.804279 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.804250 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58d70cb-6a02-44ac-b026-da52f2b1ad12-openshift-service-ca" pod "c58d70cb-6a02-44ac-b026-da52f2b1ad12" (UID: "c58d70cb-6a02-44ac-b026-da52f2b1ad12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:44.807196 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.807156 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58d70cb-6a02-44ac-b026-da52f2b1ad12-kube-api-access-42cv9" pod "c58d70cb-6a02-44ac-b026-da52f2b1ad12" (UID: "c58d70cb-6a02-44ac-b026-da52f2b1ad12"). InnerVolumeSpecName "kube-api-access-42cv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:44.807305 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.807236 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-metrics-certs" pod "c58d70cb-6a02-44ac-b026-da52f2b1ad12" (UID: "c58d70cb-6a02-44ac-b026-da52f2b1ad12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:39:44.811731 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.811704 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-maas-api-tls" pod "c58d70cb-6a02-44ac-b026-da52f2b1ad12" (UID: "c58d70cb-6a02-44ac-b026-da52f2b1ad12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:39:44.904299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.904257 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42cv9\" (UniqueName: \"kubernetes.io/projected/c58d70cb-6a02-44ac-b026-da52f2b1ad12-kube-api-access-42cv9\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.904299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.904288 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.904299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.904301 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58d70cb-6a02-44ac-b026-da52f2b1ad12-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.904566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:44.904311 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58d70cb-6a02-44ac-b026-da52f2b1ad12-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:45.530285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:45.530253 6389 generic.go:356] "Generic (PLEG): container finished" podID="c58d70cb-6a02-44ac-b026-da52f2b1ad12" containerID="14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7" exitCode=0 Nov 03 14:39:45.530754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:45.530298 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" Nov 03 14:39:45.530754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:45.530345 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" event={"ID":"c58d70cb-6a02-44ac-b026-da52f2b1ad12","Type":"ContainerDied","Data":"14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7"} Nov 03 14:39:45.530754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:45.530395 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd" event={"ID":"c58d70cb-6a02-44ac-b026-da52f2b1ad12","Type":"ContainerDied","Data":"2ed15fa561931023bb38996dee4ab07be573a1e08d52615d0dfa6bc1c9c7b4bc"} Nov 03 14:39:45.530754 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:45.530417 6389 scope.go:122] "RemoveContainer" containerID="14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7" Nov 03 14:39:45.540630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:45.540599 6389 scope.go:122] "RemoveContainer" containerID="14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7" Nov 03 14:39:45.540923 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:39:45.540900 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7\": container with ID starting with 14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7 not found: ID does not exist" containerID="14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7" Nov 03 14:39:45.540994 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:45.540935 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7"} err="failed to get container status \"14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7\": rpc error: code = NotFound desc = could not find container \"14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7\": container with ID starting with 14c743214e1495b86ccf0e1606aef53ef81e9aea6090eecaa4f88d45379b26f7 not found: ID does not exist" Nov 03 14:39:45.586415 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:45.586388 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd"] Nov 03 14:39:45.600167 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:45.600131 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-156ae1-7c74bdcff9-qbzjd"] Nov 03 14:39:46.617519 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:46.617488 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58d70cb-6a02-44ac-b026-da52f2b1ad12" path="/var/lib/kubelet/pods/c58d70cb-6a02-44ac-b026-da52f2b1ad12/volumes" Nov 03 14:39:49.788670 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:49.787684 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" podUID="142d92c8-a7e3-43fc-92bd-a8da55634087" containerName="payload-processing" containerID="cri-o://9ce4f40d09adb2f4d5fa6520f7d1bb2f809ba74501b84e0da1ca4d778b361cb9" gracePeriod=25 Nov 03 14:39:50.611753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.610547 6389 generic.go:356] "Generic (PLEG): container finished" podID="142d92c8-a7e3-43fc-92bd-a8da55634087" containerID="9ce4f40d09adb2f4d5fa6520f7d1bb2f809ba74501b84e0da1ca4d778b361cb9" exitCode=0 Nov 03 14:39:50.630746 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.618788 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" event={"ID":"142d92c8-a7e3-43fc-92bd-a8da55634087","Type":"ContainerDied","Data":"9ce4f40d09adb2f4d5fa6520f7d1bb2f809ba74501b84e0da1ca4d778b361cb9"} Nov 03 14:39:50.711178 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.711097 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:50.800030 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.799993 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-plugins-config-volume\") pod \"142d92c8-a7e3-43fc-92bd-a8da55634087\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " Nov 03 14:39:50.800591 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.800563 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-plugins-config-volume" pod "142d92c8-a7e3-43fc-92bd-a8da55634087" (UID: "142d92c8-a7e3-43fc-92bd-a8da55634087"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:50.800918 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.800647 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-kube-api-access-gxfnw\" (UniqueName: \"kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-kube-api-access-gxfnw\") pod \"142d92c8-a7e3-43fc-92bd-a8da55634087\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " Nov 03 14:39:50.803720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.802796 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-routing-overlay-volume\") pod \"142d92c8-a7e3-43fc-92bd-a8da55634087\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " Nov 03 14:39:50.803720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.802868 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-provider-credentials\" (UniqueName: \"kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-provider-credentials\") pod \"142d92c8-a7e3-43fc-92bd-a8da55634087\" (UID: \"142d92c8-a7e3-43fc-92bd-a8da55634087\") " Nov 03 14:39:50.803720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.803366 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-routing-overlay-volume" pod "142d92c8-a7e3-43fc-92bd-a8da55634087" (UID: "142d92c8-a7e3-43fc-92bd-a8da55634087"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:50.803970 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.803935 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:50.803970 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.803959 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/142d92c8-a7e3-43fc-92bd-a8da55634087-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:50.804252 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.804234 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-kube-api-access-gxfnw" pod "142d92c8-a7e3-43fc-92bd-a8da55634087" (UID: "142d92c8-a7e3-43fc-92bd-a8da55634087"). InnerVolumeSpecName "kube-api-access-gxfnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:50.805316 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.805293 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-provider-credentials" pod "142d92c8-a7e3-43fc-92bd-a8da55634087" (UID: "142d92c8-a7e3-43fc-92bd-a8da55634087"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:50.905327 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.905290 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxfnw\" (UniqueName: \"kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-kube-api-access-gxfnw\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:50.905327 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:50.905324 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/142d92c8-a7e3-43fc-92bd-a8da55634087-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:39:51.617947 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:51.617923 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" Nov 03 14:39:51.618222 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:51.617962 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4" event={"ID":"142d92c8-a7e3-43fc-92bd-a8da55634087","Type":"ContainerDied","Data":"061dfffca22c68507fc686d477b23cb0cd65ad633f7024e6cea87441f0aaa18a"} Nov 03 14:39:51.618222 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:51.618017 6389 scope.go:122] "RemoveContainer" containerID="9ce4f40d09adb2f4d5fa6520f7d1bb2f809ba74501b84e0da1ca4d778b361cb9" Nov 03 14:39:51.782970 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:51.782938 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4"] Nov 03 14:39:51.793638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:51.792856 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb49bc4f-gj7p4"] Nov 03 14:39:51.976633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:51.973857 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" podUID="1739d708-b46f-4045-862e-967f64afd54b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:39:52.412733 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.412325 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9"] Nov 03 14:39:52.413561 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.413026 6389 cpu_manager.go:414] "containerMap: removing container" podUID="142d92c8-a7e3-43fc-92bd-a8da55634087" containerName="payload-processing" Nov 03 14:39:52.413561 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.413049 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142d92c8-a7e3-43fc-92bd-a8da55634087" containerName="payload-processing" Nov 03 14:39:52.413561 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.413084 6389 cpu_manager.go:414] "containerMap: removing container" podUID="c58d70cb-6a02-44ac-b026-da52f2b1ad12" containerName="maas-api" Nov 03 14:39:52.413561 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.413092 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58d70cb-6a02-44ac-b026-da52f2b1ad12" containerName="maas-api" Nov 03 14:39:52.413561 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.413227 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="142d92c8-a7e3-43fc-92bd-a8da55634087" containerName="payload-processing" Nov 03 14:39:52.413561 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.413250 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58d70cb-6a02-44ac-b026-da52f2b1ad12" containerName="maas-api" Nov 03 14:39:52.417374 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.417348 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.450250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.450214 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9"] Nov 03 14:39:52.456816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.456784 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 14:39:52.458137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.457134 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 14:39:52.458311 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.457175 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9c7ea2cd2091\"" Nov 03 14:39:52.458408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.457255 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-zxzr2\"" Nov 03 14:39:52.458499 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.457316 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 14:39:52.458698 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.457386 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 14:39:52.508303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.508264 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-provider-credentials\") pod \"payload-processing-external-model-75cb49bc4f-m5hf9\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.508591 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.508570 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-plugins-config-volume\") pod \"payload-processing-external-model-75cb49bc4f-m5hf9\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.508803 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.508784 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgs8\" (UniqueName: \"kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-kube-api-access-ndgs8\") pod \"payload-processing-external-model-75cb49bc4f-m5hf9\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.508982 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.508965 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-routing-overlay-volume\") pod \"payload-processing-external-model-75cb49bc4f-m5hf9\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.609979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.609944 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-provider-credentials\") pod \"payload-processing-external-model-75cb49bc4f-m5hf9\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.610183 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.610014 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-plugins-config-volume\") pod \"payload-processing-external-model-75cb49bc4f-m5hf9\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.610183 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.610058 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgs8\" (UniqueName: \"kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-kube-api-access-ndgs8\") pod \"payload-processing-external-model-75cb49bc4f-m5hf9\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.610183 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.610126 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-routing-overlay-volume\") pod \"payload-processing-external-model-75cb49bc4f-m5hf9\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.611164 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.610917 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-routing-overlay-volume\") pod \"payload-processing-external-model-75cb49bc4f-m5hf9\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.611164 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.610993 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-plugins-config-volume\") pod \"payload-processing-external-model-75cb49bc4f-m5hf9\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.613217 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.613181 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-provider-credentials\") pod \"payload-processing-external-model-75cb49bc4f-m5hf9\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.617759 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.617732 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142d92c8-a7e3-43fc-92bd-a8da55634087" path="/var/lib/kubelet/pods/142d92c8-a7e3-43fc-92bd-a8da55634087/volumes" Nov 03 14:39:52.632201 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.632168 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgs8\" (UniqueName: \"kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-kube-api-access-ndgs8\") pod \"payload-processing-external-model-75cb49bc4f-m5hf9\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:52.758268 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:52.758231 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:53.088755 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:53.088729 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9"] Nov 03 14:39:53.090190 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:39:53.090163 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec145a24_1a93_46ce_823f_08565e885daa.slice/crio-00229813776c6b165766297514656726ba968eb128c3ba435e29b0f3288df054 WatchSource:0}: Error finding container 00229813776c6b165766297514656726ba968eb128c3ba435e29b0f3288df054: Status 404 returned error can't find the container with id 00229813776c6b165766297514656726ba968eb128c3ba435e29b0f3288df054 Nov 03 14:39:53.634960 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:53.634922 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" event={"ID":"ec145a24-1a93-46ce-823f-08565e885daa","Type":"ContainerStarted","Data":"5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6"} Nov 03 14:39:53.635162 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:53.634973 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" event={"ID":"ec145a24-1a93-46ce-823f-08565e885daa","Type":"ContainerStarted","Data":"00229813776c6b165766297514656726ba968eb128c3ba435e29b0f3288df054"} Nov 03 14:39:53.635162 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:53.635002 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:39:53.679192 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:53.679133 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" podStartSLOduration=1.679118991 podStartE2EDuration="1.679118991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:39:53.67869891 +0000 UTC m=+1669.237335395" watchObservedRunningTime="2026-11-03 14:39:53.679118991 +0000 UTC m=+1669.237755477" Nov 03 14:39:57.793416 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:57.793376 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9"] Nov 03 14:39:58.378905 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:39:58.378864 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:39:58.701634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.692750 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs"] Nov 03 14:39:58.705626 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.705590 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.726658 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.725298 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs"] Nov 03 14:39:58.808633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.807515 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.808633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.807632 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.808633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.807684 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.808633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.807747 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c895\" (UniqueName: \"kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-kube-api-access-6c895\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.808633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.807789 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.808633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.807842 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.808633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.807908 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.808633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.807970 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.808633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.808004 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.916812 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.916778 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.917052 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.917033 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.917249 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.917231 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.917375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.917358 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.917505 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.917491 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.917681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.917663 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c895\" (UniqueName: \"kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-kube-api-access-6c895\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.919056 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.919035 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.919254 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.919235 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.922305 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.922284 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.922526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.921100 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.923438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.920863 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.923672 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.922200 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.923794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.923391 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.923899 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.921949 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.953918 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.953827 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.966753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.966700 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.982645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.975158 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:58.988316 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:58.988239 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c895\" (UniqueName: \"kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-kube-api-access-6c895\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:59.027634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:59.024632 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:39:59.517839 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:59.517817 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs"] Nov 03 14:39:59.597131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:59.589801 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:39:59.597131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:59.589872 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:39:59.597131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:59.589899 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:39:59.715994 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:59.715960 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" event={"ID":"2a3f41f5-6f30-42b7-8d74-f6823d0abe50","Type":"ContainerStarted","Data":"a2e27a039196e7ec75f15dfd45dae31fd00146c133c5de1e2d48c297ba70a0ec"} Nov 03 14:39:59.864635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:59.843138 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:39:59.864635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:59.843259 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:39:59.969649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:59.952715 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:39:59.969649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:39:59.952783 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:40:00.854027 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:00.853981 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" event={"ID":"2a3f41f5-6f30-42b7-8d74-f6823d0abe50","Type":"ContainerStarted","Data":"24b0c7ad9510bfb8bb3b090eb5144afd8b7201fc71f20fce29ac607714b22b9b"} Nov 03 14:40:00.971633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:00.951334 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" podStartSLOduration=2.951314227 podStartE2EDuration="2.951314227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:40:00.933109448 +0000 UTC m=+1676.491745945" watchObservedRunningTime="2026-11-03 14:40:00.951314227 +0000 UTC m=+1676.509950712" Nov 03 14:40:01.031633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:01.031308 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:40:01.035631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:01.033065 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:40:01.035631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:01.033155 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" podUID="2a3f41f5-6f30-42b7-8d74-f6823d0abe50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 14:40:01.973771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:01.973727 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" podUID="1739d708-b46f-4045-862e-967f64afd54b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:40:02.035242 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.035206 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:40:02.035734 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.035701 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" podUID="2a3f41f5-6f30-42b7-8d74-f6823d0abe50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 14:40:02.081641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.081595 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth"] Nov 03 14:40:02.085416 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.085394 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.105473 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.105441 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth"] Nov 03 14:40:02.115659 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.115637 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c1f8fd209b1a\"" Nov 03 14:40:02.115824 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.115792 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mhvfr\"" Nov 03 14:40:02.177273 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.177243 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-routing-overlay-volume\") pod \"payload-processing-external-model-7cd8fc5c4-v9dth\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.177442 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.177378 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-provider-credentials\") pod \"payload-processing-external-model-7cd8fc5c4-v9dth\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.177442 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.177416 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z574j\" (UniqueName: \"kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-kube-api-access-z574j\") pod \"payload-processing-external-model-7cd8fc5c4-v9dth\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.177520 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.177502 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-plugins-config-volume\") pod \"payload-processing-external-model-7cd8fc5c4-v9dth\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.285331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.285234 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-plugins-config-volume\") pod \"payload-processing-external-model-7cd8fc5c4-v9dth\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.285634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.285590 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-routing-overlay-volume\") pod \"payload-processing-external-model-7cd8fc5c4-v9dth\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.285915 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.285895 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-provider-credentials\") pod \"payload-processing-external-model-7cd8fc5c4-v9dth\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.286063 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.286046 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z574j\" (UniqueName: \"kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-kube-api-access-z574j\") pod \"payload-processing-external-model-7cd8fc5c4-v9dth\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.287438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.287414 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-plugins-config-volume\") pod \"payload-processing-external-model-7cd8fc5c4-v9dth\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.288161 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.288140 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-routing-overlay-volume\") pod \"payload-processing-external-model-7cd8fc5c4-v9dth\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.310043 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.308724 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-provider-credentials\") pod \"payload-processing-external-model-7cd8fc5c4-v9dth\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.310043 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.309360 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z574j\" (UniqueName: \"kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-kube-api-access-z574j\") pod \"payload-processing-external-model-7cd8fc5c4-v9dth\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.399702 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.399668 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:02.740091 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.739112 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth"] Nov 03 14:40:02.749181 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:40:02.749143 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ada3b2_8595_4899_ad61_c3e42fcdf0ab.slice/crio-f66eb21804728473f68faf0c35568bf4862d31a159916c4c7f6afd0b627e2ee1 WatchSource:0}: Error finding container f66eb21804728473f68faf0c35568bf4862d31a159916c4c7f6afd0b627e2ee1: Status 404 returned error can't find the container with id f66eb21804728473f68faf0c35568bf4862d31a159916c4c7f6afd0b627e2ee1 Nov 03 14:40:02.794092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.794046 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" podUID="ec145a24-1a93-46ce-823f-08565e885daa" containerName="payload-processing" containerID="cri-o://5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6" gracePeriod=25 Nov 03 14:40:02.866365 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.866331 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" event={"ID":"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab","Type":"ContainerStarted","Data":"d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27"} Nov 03 14:40:02.866567 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.866387 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" event={"ID":"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab","Type":"ContainerStarted","Data":"f66eb21804728473f68faf0c35568bf4862d31a159916c4c7f6afd0b627e2ee1"} Nov 03 14:40:02.926638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:02.920791 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" podStartSLOduration=0.920756824 podStartE2EDuration="920.756824ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:40:02.917117205 +0000 UTC m=+1678.475753686" watchObservedRunningTime="2026-11-03 14:40:02.920756824 +0000 UTC m=+1678.479393311" Nov 03 14:40:03.029500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.028997 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:40:03.029500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.029055 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" podUID="2a3f41f5-6f30-42b7-8d74-f6823d0abe50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 14:40:03.169326 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.169302 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:40:03.204284 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.204260 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-provider-credentials\") pod \"ec145a24-1a93-46ce-823f-08565e885daa\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " Nov 03 14:40:03.204444 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.204362 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-plugins-config-volume\") pod \"ec145a24-1a93-46ce-823f-08565e885daa\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " Nov 03 14:40:03.204444 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.204415 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-routing-overlay-volume\") pod \"ec145a24-1a93-46ce-823f-08565e885daa\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " Nov 03 14:40:03.204562 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.204513 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-kube-api-access-ndgs8\" (UniqueName: \"kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-kube-api-access-ndgs8\") pod \"ec145a24-1a93-46ce-823f-08565e885daa\" (UID: \"ec145a24-1a93-46ce-823f-08565e885daa\") " Nov 03 14:40:03.204763 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.204741 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-plugins-config-volume" pod "ec145a24-1a93-46ce-823f-08565e885daa" (UID: "ec145a24-1a93-46ce-823f-08565e885daa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:40:03.204841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.204798 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-routing-overlay-volume" pod "ec145a24-1a93-46ce-823f-08565e885daa" (UID: "ec145a24-1a93-46ce-823f-08565e885daa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:40:03.204954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.204938 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:03.204989 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.204958 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec145a24-1a93-46ce-823f-08565e885daa-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:03.206495 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.206477 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-provider-credentials" pod "ec145a24-1a93-46ce-823f-08565e885daa" (UID: "ec145a24-1a93-46ce-823f-08565e885daa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:03.206723 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.206703 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-kube-api-access-ndgs8" pod "ec145a24-1a93-46ce-823f-08565e885daa" (UID: "ec145a24-1a93-46ce-823f-08565e885daa"). InnerVolumeSpecName "kube-api-access-ndgs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:03.306232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.306193 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:03.306232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.306222 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndgs8\" (UniqueName: \"kubernetes.io/projected/ec145a24-1a93-46ce-823f-08565e885daa-kube-api-access-ndgs8\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:03.873147 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.873117 6389 generic.go:356] "Generic (PLEG): container finished" podID="ec145a24-1a93-46ce-823f-08565e885daa" containerID="5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6" exitCode=0 Nov 03 14:40:03.873331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.873162 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" Nov 03 14:40:03.873331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.873205 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" event={"ID":"ec145a24-1a93-46ce-823f-08565e885daa","Type":"ContainerDied","Data":"5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6"} Nov 03 14:40:03.873331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.873254 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9" event={"ID":"ec145a24-1a93-46ce-823f-08565e885daa","Type":"ContainerDied","Data":"00229813776c6b165766297514656726ba968eb128c3ba435e29b0f3288df054"} Nov 03 14:40:03.873331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.873282 6389 scope.go:122] "RemoveContainer" containerID="5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6" Nov 03 14:40:03.873650 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.873587 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:03.885096 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.885074 6389 scope.go:122] "RemoveContainer" containerID="5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6" Nov 03 14:40:03.885558 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:40:03.885540 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6\": container with ID starting with 5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6 not found: ID does not exist" containerID="5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6" Nov 03 14:40:03.885674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.885568 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6"} err="failed to get container status \"5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6\": rpc error: code = NotFound desc = could not find container \"5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6\": container with ID starting with 5dbb2722709726ee5cc637596f92b44a44b5d676c19d9f2895d6a4d1a1d63ae6 not found: ID does not exist" Nov 03 14:40:03.945109 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.945075 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9"] Nov 03 14:40:03.950537 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:03.950515 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb49bc4f-m5hf9"] Nov 03 14:40:04.038634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:04.031000 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:40:04.038634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:04.031058 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" podUID="2a3f41f5-6f30-42b7-8d74-f6823d0abe50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 14:40:04.617417 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:04.617383 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec145a24-1a93-46ce-823f-08565e885daa" path="/var/lib/kubelet/pods/ec145a24-1a93-46ce-823f-08565e885daa/volumes" Nov 03 14:40:05.042703 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.042211 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:40:05.042703 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.042280 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" podUID="2a3f41f5-6f30-42b7-8d74-f6823d0abe50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 14:40:05.075120 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.075087 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" Nov 03 14:40:05.151751 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.151662 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1739d708-b46f-4045-862e-967f64afd54b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1739d708-b46f-4045-862e-967f64afd54b-plugins-config-volume\") pod \"1739d708-b46f-4045-862e-967f64afd54b\" (UID: \"1739d708-b46f-4045-862e-967f64afd54b\") " Nov 03 14:40:05.151932 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.151772 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1739d708-b46f-4045-862e-967f64afd54b-kube-api-access-qqmq9\" (UniqueName: \"kubernetes.io/projected/1739d708-b46f-4045-862e-967f64afd54b-kube-api-access-qqmq9\") pod \"1739d708-b46f-4045-862e-967f64afd54b\" (UID: \"1739d708-b46f-4045-862e-967f64afd54b\") " Nov 03 14:40:05.152879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.152839 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1739d708-b46f-4045-862e-967f64afd54b-plugins-config-volume" pod "1739d708-b46f-4045-862e-967f64afd54b" (UID: "1739d708-b46f-4045-862e-967f64afd54b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:40:05.165533 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.164764 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1739d708-b46f-4045-862e-967f64afd54b-kube-api-access-qqmq9" pod "1739d708-b46f-4045-862e-967f64afd54b" (UID: "1739d708-b46f-4045-862e-967f64afd54b"). InnerVolumeSpecName "kube-api-access-qqmq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:05.253179 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.253139 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1739d708-b46f-4045-862e-967f64afd54b-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:05.253179 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.253171 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqmq9\" (UniqueName: \"kubernetes.io/projected/1739d708-b46f-4045-862e-967f64afd54b-kube-api-access-qqmq9\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:05.899082 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.899044 6389 generic.go:356] "Generic (PLEG): container finished" podID="1739d708-b46f-4045-862e-967f64afd54b" containerID="b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b" exitCode=0 Nov 03 14:40:05.899303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.899143 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" Nov 03 14:40:05.899303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.899201 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" event={"ID":"1739d708-b46f-4045-862e-967f64afd54b","Type":"ContainerDied","Data":"b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b"} Nov 03 14:40:05.899303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.899255 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz" event={"ID":"1739d708-b46f-4045-862e-967f64afd54b","Type":"ContainerDied","Data":"e0e0cf4dee14d74ab970a6568162f717dba9a1557949a5d0290e82c2562828fe"} Nov 03 14:40:05.899303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.899282 6389 scope.go:122] "RemoveContainer" containerID="b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b" Nov 03 14:40:05.912682 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.912661 6389 scope.go:122] "RemoveContainer" containerID="b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b" Nov 03 14:40:05.913121 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:40:05.913099 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b\": container with ID starting with b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b not found: ID does not exist" containerID="b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b" Nov 03 14:40:05.913237 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.913127 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b"} err="failed to get container status \"b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b\": rpc error: code = NotFound desc = could not find container \"b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b\": container with ID starting with b5cafc3a2cc7b7dd0476ec661b7f4043ef212b5023b457c780b43e3dec70b46b not found: ID does not exist" Nov 03 14:40:05.954322 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.954291 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz"] Nov 03 14:40:05.966060 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:05.966034 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-156ae1-7df59f7959-j7jdz"] Nov 03 14:40:06.040635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:06.039948 6389 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:40:06.040635 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:06.040025 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" podUID="2a3f41f5-6f30-42b7-8d74-f6823d0abe50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 14:40:06.495571 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:06.495523 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth"] Nov 03 14:40:06.620375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:06.620341 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1739d708-b46f-4045-862e-967f64afd54b" path="/var/lib/kubelet/pods/1739d708-b46f-4045-862e-967f64afd54b/volumes" Nov 03 14:40:07.029711 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:07.029674 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:40:07.030106 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:07.030076 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:40:07.031332 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:07.031312 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:40:11.496769 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.496723 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" podUID="c1ada3b2-8595-4899-ad61-c3e42fcdf0ab" containerName="payload-processing" containerID="cri-o://d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27" gracePeriod=25 Nov 03 14:40:11.780687 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.780654 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4"] Nov 03 14:40:11.780945 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.780893 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" podUID="5004ddef-5dbd-4ea1-bb10-bc82a37791b8" containerName="istio-proxy" containerID="cri-o://0b84f7b3ca712899e9d4406b06f1f91da610a51f1bc5c91327035fc37cbeb245" gracePeriod=30 Nov 03 14:40:11.786550 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.786512 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:11.793340 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.793320 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc"] Nov 03 14:40:11.793533 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.793493 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" podUID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" containerName="istio-proxy" containerID="cri-o://917af16a05d736cb6a02b1140ceda5c4aff93ba42769990c370303e4a9444a85" gracePeriod=30 Nov 03 14:40:11.795851 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.795828 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs"] Nov 03 14:40:11.796032 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.796007 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" podUID="2a3f41f5-6f30-42b7-8d74-f6823d0abe50" containerName="istio-proxy" containerID="cri-o://24b0c7ad9510bfb8bb3b090eb5144afd8b7201fc71f20fce29ac607714b22b9b" gracePeriod=30 Nov 03 14:40:11.841675 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.841633 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-routing-overlay-volume\") pod \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " Nov 03 14:40:11.841845 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.841684 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-provider-credentials\") pod \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " Nov 03 14:40:11.841845 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.841743 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-plugins-config-volume\") pod \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " Nov 03 14:40:11.841845 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.841820 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-kube-api-access-z574j\" (UniqueName: \"kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-kube-api-access-z574j\") pod \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\" (UID: \"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab\") " Nov 03 14:40:11.842110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.842078 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-routing-overlay-volume" pod "c1ada3b2-8595-4899-ad61-c3e42fcdf0ab" (UID: "c1ada3b2-8595-4899-ad61-c3e42fcdf0ab"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:40:11.842212 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.842193 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-plugins-config-volume" pod "c1ada3b2-8595-4899-ad61-c3e42fcdf0ab" (UID: "c1ada3b2-8595-4899-ad61-c3e42fcdf0ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:40:11.842269 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.842205 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:11.844029 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.844002 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-provider-credentials" pod "c1ada3b2-8595-4899-ad61-c3e42fcdf0ab" (UID: "c1ada3b2-8595-4899-ad61-c3e42fcdf0ab"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:11.844140 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.844008 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-kube-api-access-z574j" pod "c1ada3b2-8595-4899-ad61-c3e42fcdf0ab" (UID: "c1ada3b2-8595-4899-ad61-c3e42fcdf0ab"). InnerVolumeSpecName "kube-api-access-z574j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:11.943421 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.943380 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:11.943421 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.943411 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:11.943421 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.943424 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z574j\" (UniqueName: \"kubernetes.io/projected/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab-kube-api-access-z574j\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:11.990166 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.990133 6389 generic.go:356] "Generic (PLEG): container finished" podID="c1ada3b2-8595-4899-ad61-c3e42fcdf0ab" containerID="d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27" exitCode=0 Nov 03 14:40:11.990314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.990175 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" Nov 03 14:40:11.990314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.990193 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" event={"ID":"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab","Type":"ContainerDied","Data":"d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27"} Nov 03 14:40:11.990314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.990251 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth" event={"ID":"c1ada3b2-8595-4899-ad61-c3e42fcdf0ab","Type":"ContainerDied","Data":"f66eb21804728473f68faf0c35568bf4862d31a159916c4c7f6afd0b627e2ee1"} Nov 03 14:40:11.990314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:11.990275 6389 scope.go:122] "RemoveContainer" containerID="d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27" Nov 03 14:40:12.000481 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.000465 6389 scope.go:122] "RemoveContainer" containerID="d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27" Nov 03 14:40:12.000798 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:40:12.000778 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27\": container with ID starting with d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27 not found: ID does not exist" containerID="d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27" Nov 03 14:40:12.000874 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.000811 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27"} err="failed to get container status \"d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27\": rpc error: code = NotFound desc = could not find container \"d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27\": container with ID starting with d880050b3510a134e9645b4cc84fe0e33974f9f21e0d8c9a9efb7942abd35b27 not found: ID does not exist" Nov 03 14:40:12.049638 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.049473 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth"] Nov 03 14:40:12.052200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.052174 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd8fc5c4-v9dth"] Nov 03 14:40:12.498712 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.498685 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z"] Nov 03 14:40:12.499190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.499175 6389 cpu_manager.go:414] "containerMap: removing container" podUID="c1ada3b2-8595-4899-ad61-c3e42fcdf0ab" containerName="payload-processing" Nov 03 14:40:12.499232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.499192 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ada3b2-8595-4899-ad61-c3e42fcdf0ab" containerName="payload-processing" Nov 03 14:40:12.499232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.499201 6389 cpu_manager.go:414] "containerMap: removing container" podUID="1739d708-b46f-4045-862e-967f64afd54b" containerName="payload-pre-processing" Nov 03 14:40:12.499232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.499207 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1739d708-b46f-4045-862e-967f64afd54b" containerName="payload-pre-processing" Nov 03 14:40:12.499341 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.499240 6389 cpu_manager.go:414] "containerMap: removing container" podUID="ec145a24-1a93-46ce-823f-08565e885daa" containerName="payload-processing" Nov 03 14:40:12.499341 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.499245 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec145a24-1a93-46ce-823f-08565e885daa" containerName="payload-processing" Nov 03 14:40:12.499341 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.499315 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec145a24-1a93-46ce-823f-08565e885daa" containerName="payload-processing" Nov 03 14:40:12.499341 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.499327 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ada3b2-8595-4899-ad61-c3e42fcdf0ab" containerName="payload-processing" Nov 03 14:40:12.499341 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.499334 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="1739d708-b46f-4045-862e-967f64afd54b" containerName="payload-pre-processing" Nov 03 14:40:12.503896 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.503872 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.524145 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.524119 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-6e0eeb210ae0\"" Nov 03 14:40:12.524359 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.524342 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-e2e--e9e6f325-dockercfg-mxcpf\"" Nov 03 14:40:12.524665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.524644 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"routing-overlay\"" Nov 03 14:40:12.525482 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.525458 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z"] Nov 03 14:40:12.532972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.532948 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-331c9e\"" Nov 03 14:40:12.532972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.532963 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"kube-root-ca.crt\"" Nov 03 14:40:12.532972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.532966 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"openshift-service-ca.crt\"" Nov 03 14:40:12.551912 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.551875 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.551912 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.551932 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.552138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.552038 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.552138 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.552084 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8js\" (UniqueName: \"kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-kube-api-access-2c8js\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.617767 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.617733 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ada3b2-8595-4899-ad61-c3e42fcdf0ab" path="/var/lib/kubelet/pods/c1ada3b2-8595-4899-ad61-c3e42fcdf0ab/volumes" Nov 03 14:40:12.655631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.653138 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.655631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.653207 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.655631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.653266 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.655631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.653321 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8js\" (UniqueName: \"kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-kube-api-access-2c8js\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.655631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.654582 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.656043 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.655703 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.675221 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.674692 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.680958 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.680746 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8js\" (UniqueName: \"kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-kube-api-access-2c8js\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:12.819756 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:12.819670 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:13.139012 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:13.138947 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z"] Nov 03 14:40:13.140728 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:40:13.140696 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5304dea_ff1c_4421_8116_2ac7fe91ef73.slice/crio-13ec85fe8676eb15246424e7c939511a4d9d88d80c73ccd6ae0c14edcd988712 WatchSource:0}: Error finding container 13ec85fe8676eb15246424e7c939511a4d9d88d80c73ccd6ae0c14edcd988712: Status 404 returned error can't find the container with id 13ec85fe8676eb15246424e7c939511a4d9d88d80c73ccd6ae0c14edcd988712 Nov 03 14:40:14.003551 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:14.003512 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" event={"ID":"e5304dea-ff1c-4421-8116-2ac7fe91ef73","Type":"ContainerStarted","Data":"873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40"} Nov 03 14:40:14.003988 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:14.003624 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" event={"ID":"e5304dea-ff1c-4421-8116-2ac7fe91ef73","Type":"ContainerStarted","Data":"13ec85fe8676eb15246424e7c939511a4d9d88d80c73ccd6ae0c14edcd988712"} Nov 03 14:40:14.003988 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:14.003732 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:14.052959 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:14.052907 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" podStartSLOduration=2.052894989 podStartE2EDuration="2.052894989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:40:14.051848578 +0000 UTC m=+1689.610485062" watchObservedRunningTime="2026-11-03 14:40:14.052894989 +0000 UTC m=+1689.611531472" Nov 03 14:40:17.021794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.021765 6389 generic.go:356] "Generic (PLEG): container finished" podID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" containerID="917af16a05d736cb6a02b1140ceda5c4aff93ba42769990c370303e4a9444a85" exitCode=0 Nov 03 14:40:17.022230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.021824 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" event={"ID":"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5","Type":"ContainerDied","Data":"917af16a05d736cb6a02b1140ceda5c4aff93ba42769990c370303e4a9444a85"} Nov 03 14:40:17.023776 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.023733 6389 generic.go:356] "Generic (PLEG): container finished" podID="5004ddef-5dbd-4ea1-bb10-bc82a37791b8" containerID="0b84f7b3ca712899e9d4406b06f1f91da610a51f1bc5c91327035fc37cbeb245" exitCode=0 Nov 03 14:40:17.023914 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.023807 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" event={"ID":"5004ddef-5dbd-4ea1-bb10-bc82a37791b8","Type":"ContainerDied","Data":"0b84f7b3ca712899e9d4406b06f1f91da610a51f1bc5c91327035fc37cbeb245"} Nov 03 14:40:17.025781 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.025758 6389 generic.go:356] "Generic (PLEG): container finished" podID="2a3f41f5-6f30-42b7-8d74-f6823d0abe50" containerID="24b0c7ad9510bfb8bb3b090eb5144afd8b7201fc71f20fce29ac607714b22b9b" exitCode=0 Nov 03 14:40:17.025873 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.025834 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" event={"ID":"2a3f41f5-6f30-42b7-8d74-f6823d0abe50","Type":"ContainerDied","Data":"24b0c7ad9510bfb8bb3b090eb5144afd8b7201fc71f20fce29ac607714b22b9b"} Nov 03 14:40:17.061905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.061832 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:40:17.105586 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.105558 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:40:17.110179 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.110152 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:40:17.205086 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205058 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-socket\") pod \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " Nov 03 14:40:17.205235 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205107 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-envoy\") pod \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " Nov 03 14:40:17.205235 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205124 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-envoy\") pod \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " Nov 03 14:40:17.205235 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205166 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-socket\") pod \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " Nov 03 14:40:17.205235 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205188 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-podinfo\") pod \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " Nov 03 14:40:17.205235 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205228 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-token\" (UniqueName: \"kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-token\") pod \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " Nov 03 14:40:17.205691 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205666 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-certs\") pod \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " Nov 03 14:40:17.205829 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205706 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-credential-socket\") pod \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " Nov 03 14:40:17.205829 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205739 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-credential-socket\") pod \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " Nov 03 14:40:17.205829 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205785 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-socket\") pod \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " Nov 03 14:40:17.205829 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205817 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-certs\") pod \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " Nov 03 14:40:17.206131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205864 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-envoy\") pod \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " Nov 03 14:40:17.206131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205912 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istiod-ca-cert\") pod \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " Nov 03 14:40:17.206131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205978 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-credential-socket\") pod \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " Nov 03 14:40:17.206131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206013 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-podinfo\") pod \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " Nov 03 14:40:17.206131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206057 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-kube-api-access-6c895\" (UniqueName: \"kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-kube-api-access-6c895\") pod \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " Nov 03 14:40:17.206131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206086 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istiod-ca-cert\") pod \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " Nov 03 14:40:17.206131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205407 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-socket" pod "2a3f41f5-6f30-42b7-8d74-f6823d0abe50" (UID: "2a3f41f5-6f30-42b7-8d74-f6823d0abe50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.206131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206117 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-data\") pod \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " Nov 03 14:40:17.206535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205453 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-socket" pod "5004ddef-5dbd-4ea1-bb10-bc82a37791b8" (UID: "5004ddef-5dbd-4ea1-bb10-bc82a37791b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.206535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.205901 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-certs" pod "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" (UID: "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.206535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206144 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-data\") pod \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " Nov 03 14:40:17.206535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206180 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-kube-api-access-k7m8n\" (UniqueName: \"kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-kube-api-access-k7m8n\") pod \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " Nov 03 14:40:17.206535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206217 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-token\" (UniqueName: \"kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-token\") pod \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\" (UID: \"2a3f41f5-6f30-42b7-8d74-f6823d0abe50\") " Nov 03 14:40:17.206535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206269 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istiod-ca-cert\") pod \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " Nov 03 14:40:17.206535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206303 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-certs\") pod \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " Nov 03 14:40:17.206535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206346 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-kube-api-access-8d9km\" (UniqueName: \"kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-kube-api-access-8d9km\") pod \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\" (UID: \"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5\") " Nov 03 14:40:17.206939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206375 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-podinfo\") pod \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " Nov 03 14:40:17.206939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206713 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-data\") pod \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " Nov 03 14:40:17.206939 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206741 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-token\" (UniqueName: \"kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-token\") pod \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\" (UID: \"5004ddef-5dbd-4ea1-bb10-bc82a37791b8\") " Nov 03 14:40:17.207217 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.207200 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.207285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.207221 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.207285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.207237 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.207993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206107 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-credential-socket" pod "5004ddef-5dbd-4ea1-bb10-bc82a37791b8" (UID: "5004ddef-5dbd-4ea1-bb10-bc82a37791b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.207993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206307 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-credential-socket" pod "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" (UID: "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.207993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206601 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-data" pod "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" (UID: "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.207993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206811 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-data" pod "2a3f41f5-6f30-42b7-8d74-f6823d0abe50" (UID: "2a3f41f5-6f30-42b7-8d74-f6823d0abe50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.207993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.206993 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-credential-socket" pod "2a3f41f5-6f30-42b7-8d74-f6823d0abe50" (UID: "2a3f41f5-6f30-42b7-8d74-f6823d0abe50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.207993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.207013 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istiod-ca-cert" pod "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" (UID: "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:40:17.207993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.207192 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-certs" pod "5004ddef-5dbd-4ea1-bb10-bc82a37791b8" (UID: "5004ddef-5dbd-4ea1-bb10-bc82a37791b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.207993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.207273 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istiod-ca-cert" pod "2a3f41f5-6f30-42b7-8d74-f6823d0abe50" (UID: "2a3f41f5-6f30-42b7-8d74-f6823d0abe50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:40:17.207993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.207623 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istiod-ca-cert" pod "5004ddef-5dbd-4ea1-bb10-bc82a37791b8" (UID: "5004ddef-5dbd-4ea1-bb10-bc82a37791b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:40:17.207993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.207953 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-socket" pod "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" (UID: "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.208681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.208054 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-data" pod "5004ddef-5dbd-4ea1-bb10-bc82a37791b8" (UID: "5004ddef-5dbd-4ea1-bb10-bc82a37791b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.208681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.208150 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-certs" pod "2a3f41f5-6f30-42b7-8d74-f6823d0abe50" (UID: "2a3f41f5-6f30-42b7-8d74-f6823d0abe50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.209018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.208975 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-podinfo" pod "2a3f41f5-6f30-42b7-8d74-f6823d0abe50" (UID: "2a3f41f5-6f30-42b7-8d74-f6823d0abe50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:40:17.209018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.208980 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-envoy" pod "5004ddef-5dbd-4ea1-bb10-bc82a37791b8" (UID: "5004ddef-5dbd-4ea1-bb10-bc82a37791b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.209171 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.209007 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-envoy" pod "2a3f41f5-6f30-42b7-8d74-f6823d0abe50" (UID: "2a3f41f5-6f30-42b7-8d74-f6823d0abe50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.210155 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.210129 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-token" pod "2a3f41f5-6f30-42b7-8d74-f6823d0abe50" (UID: "2a3f41f5-6f30-42b7-8d74-f6823d0abe50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:17.210886 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.210855 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-token" pod "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" (UID: "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:17.210976 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.210856 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-podinfo" pod "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" (UID: "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:40:17.211321 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.211296 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-kube-api-access-8d9km" pod "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" (UID: "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5"). InnerVolumeSpecName "kube-api-access-8d9km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:17.211452 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.211435 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-kube-api-access-k7m8n" pod "5004ddef-5dbd-4ea1-bb10-bc82a37791b8" (UID: "5004ddef-5dbd-4ea1-bb10-bc82a37791b8"). InnerVolumeSpecName "kube-api-access-k7m8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:17.211505 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.211442 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-podinfo" pod "5004ddef-5dbd-4ea1-bb10-bc82a37791b8" (UID: "5004ddef-5dbd-4ea1-bb10-bc82a37791b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:40:17.211806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.211786 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-token" pod "5004ddef-5dbd-4ea1-bb10-bc82a37791b8" (UID: "5004ddef-5dbd-4ea1-bb10-bc82a37791b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:17.211879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.211815 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-envoy" pod "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" (UID: "ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.212187 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.212168 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-kube-api-access-6c895" pod "2a3f41f5-6f30-42b7-8d74-f6823d0abe50" (UID: "2a3f41f5-6f30-42b7-8d74-f6823d0abe50"). InnerVolumeSpecName "kube-api-access-6c895". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:17.308177 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308144 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308177 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308175 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308189 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308202 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308214 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308225 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308235 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308246 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308257 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308269 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c895\" (UniqueName: \"kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-kube-api-access-6c895\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308281 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308294 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308304 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308315 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7m8n\" (UniqueName: \"kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-kube-api-access-k7m8n\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308327 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308339 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308349 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308360 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d9km\" (UniqueName: \"kubernetes.io/projected/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5-kube-api-access-8d9km\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308375 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308389 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308402 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308414 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.308424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308425 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5004ddef-5dbd-4ea1-bb10-bc82a37791b8-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.309081 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:17.308437 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a3f41f5-6f30-42b7-8d74-f6823d0abe50-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:40:18.032405 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.032374 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" Nov 03 14:40:18.032925 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.032404 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs" event={"ID":"2a3f41f5-6f30-42b7-8d74-f6823d0abe50","Type":"ContainerDied","Data":"a2e27a039196e7ec75f15dfd45dae31fd00146c133c5de1e2d48c297ba70a0ec"} Nov 03 14:40:18.032925 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.032464 6389 scope.go:122] "RemoveContainer" containerID="24b0c7ad9510bfb8bb3b090eb5144afd8b7201fc71f20fce29ac607714b22b9b" Nov 03 14:40:18.034029 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.034001 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" Nov 03 14:40:18.034155 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.034130 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc" event={"ID":"ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5","Type":"ContainerDied","Data":"54a55b3a7b5e02c2d5233d1ffb92f4004e7ace6dd2b7bb89124f983558225f96"} Nov 03 14:40:18.035706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.035683 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" event={"ID":"5004ddef-5dbd-4ea1-bb10-bc82a37791b8","Type":"ContainerDied","Data":"691f7779ce6a23e715d332d1524355f9770c3ab7be4e9381aa8017c2be06f27a"} Nov 03 14:40:18.035828 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.035728 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4" Nov 03 14:40:18.042037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.042020 6389 scope.go:122] "RemoveContainer" containerID="917af16a05d736cb6a02b1140ceda5c4aff93ba42769990c370303e4a9444a85" Nov 03 14:40:18.051518 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.051496 6389 scope.go:122] "RemoveContainer" containerID="0b84f7b3ca712899e9d4406b06f1f91da610a51f1bc5c91327035fc37cbeb245" Nov 03 14:40:18.100305 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.100268 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc"] Nov 03 14:40:18.112715 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.112688 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-c7vhc"] Nov 03 14:40:18.149854 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.149829 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs"] Nov 03 14:40:18.156517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.156490 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-thrxs"] Nov 03 14:40:18.196603 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.196569 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4"] Nov 03 14:40:18.203088 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.203062 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-wphv4"] Nov 03 14:40:18.618936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.618900 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a3f41f5-6f30-42b7-8d74-f6823d0abe50" path="/var/lib/kubelet/pods/2a3f41f5-6f30-42b7-8d74-f6823d0abe50/volumes" Nov 03 14:40:18.619352 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.619338 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5004ddef-5dbd-4ea1-bb10-bc82a37791b8" path="/var/lib/kubelet/pods/5004ddef-5dbd-4ea1-bb10-bc82a37791b8/volumes" Nov 03 14:40:18.619756 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:18.619744 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" path="/var/lib/kubelet/pods/ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5/volumes" Nov 03 14:40:25.012147 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:25.012114 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:40:28.373492 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:40:28.373452 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:40:35.411464 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.411426 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2"] Nov 03 14:40:35.413966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.411899 6389 cpu_manager.go:414] "containerMap: removing container" podUID="2a3f41f5-6f30-42b7-8d74-f6823d0abe50" containerName="istio-proxy" Nov 03 14:40:35.413966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.411910 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a3f41f5-6f30-42b7-8d74-f6823d0abe50" containerName="istio-proxy" Nov 03 14:40:35.413966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.411923 6389 cpu_manager.go:414] "containerMap: removing container" podUID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" containerName="istio-proxy" Nov 03 14:40:35.413966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.411929 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" containerName="istio-proxy" Nov 03 14:40:35.413966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.411937 6389 cpu_manager.go:414] "containerMap: removing container" podUID="5004ddef-5dbd-4ea1-bb10-bc82a37791b8" containerName="istio-proxy" Nov 03 14:40:35.413966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.411942 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5004ddef-5dbd-4ea1-bb10-bc82a37791b8" containerName="istio-proxy" Nov 03 14:40:35.413966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.411999 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a3f41f5-6f30-42b7-8d74-f6823d0abe50" containerName="istio-proxy" Nov 03 14:40:35.413966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.412008 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1b2c5f-ad7b-4a8b-a6b7-6b0082106ca5" containerName="istio-proxy" Nov 03 14:40:35.413966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.412014 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="5004ddef-5dbd-4ea1-bb10-bc82a37791b8" containerName="istio-proxy" Nov 03 14:40:35.415024 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.415006 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.438411 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.438384 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-96d371a313d1\"" Nov 03 14:40:35.455652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.450001 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"openshift-service-ca.crt\"" Nov 03 14:40:35.455652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.450012 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"routing-overlay\"" Nov 03 14:40:35.455652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.450012 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b4361f\"" Nov 03 14:40:35.455652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.450051 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"kube-root-ca.crt\"" Nov 03 14:40:35.455652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.452929 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2"] Nov 03 14:40:35.535168 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.535111 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.535362 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.535188 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.535362 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.535228 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvzq8\" (UniqueName: \"kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-kube-api-access-wvzq8\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.535440 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.535357 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.535815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.535800 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-e2e--29c79c46-dockercfg-ldkkf\"" Nov 03 14:40:35.636698 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.636667 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.636698 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.636699 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzq8\" (UniqueName: \"kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-kube-api-access-wvzq8\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.636963 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.636761 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.636963 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.636844 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.637595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.637571 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.637695 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.637669 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.639573 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.639551 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.660050 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.660022 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzq8\" (UniqueName: \"kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-kube-api-access-wvzq8\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:35.727951 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:35.727912 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:36.032581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:36.032554 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2"] Nov 03 14:40:36.034193 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:40:36.034165 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6d1f9b_4748_4a26_bd59_89a17586eeaf.slice/crio-a783a2c87ef348ec39402b1fb05bbf9098b450f1a844c105b46433d6af45f66d WatchSource:0}: Error finding container a783a2c87ef348ec39402b1fb05bbf9098b450f1a844c105b46433d6af45f66d: Status 404 returned error can't find the container with id a783a2c87ef348ec39402b1fb05bbf9098b450f1a844c105b46433d6af45f66d Nov 03 14:40:36.131114 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:36.131078 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" event={"ID":"be6d1f9b-4748-4a26-bd59-89a17586eeaf","Type":"ContainerStarted","Data":"e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5"} Nov 03 14:40:36.131283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:36.131129 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" event={"ID":"be6d1f9b-4748-4a26-bd59-89a17586eeaf","Type":"ContainerStarted","Data":"a783a2c87ef348ec39402b1fb05bbf9098b450f1a844c105b46433d6af45f66d"} Nov 03 14:40:36.131283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:36.131186 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:36.186229 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:36.186174 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" podStartSLOduration=1.186158665 podStartE2EDuration="1.186158665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:40:36.185692512 +0000 UTC m=+1711.744328993" watchObservedRunningTime="2026-11-03 14:40:36.186158665 +0000 UTC m=+1711.744795148" Nov 03 14:40:47.139753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:40:47.139663 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:40:58.373078 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:40:58.373042 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:41:07.188063 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:07.188031 6389 scope.go:122] "RemoveContainer" containerID="2868d259b9a2cfbccdc8c58fd8223eaa2155ca1f81fd6a0156238b2926dbaae9" Nov 03 14:41:11.491663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:11.491627 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2"] Nov 03 14:41:16.492685 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.492642 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" podUID="be6d1f9b-4748-4a26-bd59-89a17586eeaf" containerName="payload-processing" containerID="cri-o://e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5" gracePeriod=25 Nov 03 14:41:16.749048 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.748988 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:41:16.893052 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.893020 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-kube-api-access-wvzq8\" (UniqueName: \"kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-kube-api-access-wvzq8\") pod \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " Nov 03 14:41:16.893052 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.893058 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-routing-overlay-volume\") pod \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " Nov 03 14:41:16.893314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.893135 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-provider-credentials\") pod \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " Nov 03 14:41:16.893314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.893177 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-plugins-config-volume\") pod \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\" (UID: \"be6d1f9b-4748-4a26-bd59-89a17586eeaf\") " Nov 03 14:41:16.893508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.893484 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-routing-overlay-volume" pod "be6d1f9b-4748-4a26-bd59-89a17586eeaf" (UID: "be6d1f9b-4748-4a26-bd59-89a17586eeaf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:41:16.893645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.893593 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-plugins-config-volume" pod "be6d1f9b-4748-4a26-bd59-89a17586eeaf" (UID: "be6d1f9b-4748-4a26-bd59-89a17586eeaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:41:16.895334 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.895313 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-kube-api-access-wvzq8" pod "be6d1f9b-4748-4a26-bd59-89a17586eeaf" (UID: "be6d1f9b-4748-4a26-bd59-89a17586eeaf"). InnerVolumeSpecName "kube-api-access-wvzq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:41:16.895412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.895354 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-provider-credentials" pod "be6d1f9b-4748-4a26-bd59-89a17586eeaf" (UID: "be6d1f9b-4748-4a26-bd59-89a17586eeaf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:41:16.994456 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.994426 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvzq8\" (UniqueName: \"kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-kube-api-access-wvzq8\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:41:16.994456 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.994453 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:41:16.994456 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.994462 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be6d1f9b-4748-4a26-bd59-89a17586eeaf-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:41:16.994709 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:16.994470 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6d1f9b-4748-4a26-bd59-89a17586eeaf-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:41:17.336174 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:17.336142 6389 generic.go:356] "Generic (PLEG): container finished" podID="be6d1f9b-4748-4a26-bd59-89a17586eeaf" containerID="e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5" exitCode=0 Nov 03 14:41:17.336375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:17.336188 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" Nov 03 14:41:17.336375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:17.336230 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" event={"ID":"be6d1f9b-4748-4a26-bd59-89a17586eeaf","Type":"ContainerDied","Data":"e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5"} Nov 03 14:41:17.336375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:17.336307 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2" event={"ID":"be6d1f9b-4748-4a26-bd59-89a17586eeaf","Type":"ContainerDied","Data":"a783a2c87ef348ec39402b1fb05bbf9098b450f1a844c105b46433d6af45f66d"} Nov 03 14:41:17.336375 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:17.336331 6389 scope.go:122] "RemoveContainer" containerID="e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5" Nov 03 14:41:17.345821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:17.345806 6389 scope.go:122] "RemoveContainer" containerID="e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5" Nov 03 14:41:17.346092 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:41:17.346074 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5\": container with ID starting with e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5 not found: ID does not exist" containerID="e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5" Nov 03 14:41:17.346165 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:17.346128 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5"} err="failed to get container status \"e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5\": rpc error: code = NotFound desc = could not find container \"e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5\": container with ID starting with e45d3d6766beb89393527b2184d6236773b7e2505bf14adc693150f8baf774f5 not found: ID does not exist" Nov 03 14:41:17.397133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:17.397104 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2"] Nov 03 14:41:17.409892 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:17.409868 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5b79fgkp2"] Nov 03 14:41:18.617651 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:18.617596 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6d1f9b-4748-4a26-bd59-89a17586eeaf" path="/var/lib/kubelet/pods/be6d1f9b-4748-4a26-bd59-89a17586eeaf/volumes" Nov 03 14:41:23.605647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:23.589533 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z"] Nov 03 14:41:28.373804 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:41:28.373761 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:41:28.604793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:28.604727 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" podUID="e5304dea-ff1c-4421-8116-2ac7fe91ef73" containerName="payload-processing" containerID="cri-o://873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40" gracePeriod=25 Nov 03 14:41:28.859372 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:28.859348 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:41:29.033730 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.033697 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-routing-overlay-volume\") pod \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " Nov 03 14:41:29.033910 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.033757 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-kube-api-access-2c8js\" (UniqueName: \"kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-kube-api-access-2c8js\") pod \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " Nov 03 14:41:29.033910 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.033807 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-plugins-config-volume\") pod \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " Nov 03 14:41:29.033910 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.033867 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-provider-credentials\") pod \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\" (UID: \"e5304dea-ff1c-4421-8116-2ac7fe91ef73\") " Nov 03 14:41:29.034092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.034065 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-routing-overlay-volume" pod "e5304dea-ff1c-4421-8116-2ac7fe91ef73" (UID: "e5304dea-ff1c-4421-8116-2ac7fe91ef73"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:41:29.034232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.034211 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-plugins-config-volume" pod "e5304dea-ff1c-4421-8116-2ac7fe91ef73" (UID: "e5304dea-ff1c-4421-8116-2ac7fe91ef73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:41:29.034311 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.034297 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:41:29.034370 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.034313 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5304dea-ff1c-4421-8116-2ac7fe91ef73-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:41:29.036121 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.036095 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-provider-credentials" pod "e5304dea-ff1c-4421-8116-2ac7fe91ef73" (UID: "e5304dea-ff1c-4421-8116-2ac7fe91ef73"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:41:29.036249 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.036153 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-kube-api-access-2c8js" pod "e5304dea-ff1c-4421-8116-2ac7fe91ef73" (UID: "e5304dea-ff1c-4421-8116-2ac7fe91ef73"). InnerVolumeSpecName "kube-api-access-2c8js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:41:29.134979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.134946 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:41:29.134979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.134977 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c8js\" (UniqueName: \"kubernetes.io/projected/e5304dea-ff1c-4421-8116-2ac7fe91ef73-kube-api-access-2c8js\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:41:29.405396 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.405318 6389 generic.go:356] "Generic (PLEG): container finished" podID="e5304dea-ff1c-4421-8116-2ac7fe91ef73" containerID="873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40" exitCode=0 Nov 03 14:41:29.405396 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.405363 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" Nov 03 14:41:29.406018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.405402 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" event={"ID":"e5304dea-ff1c-4421-8116-2ac7fe91ef73","Type":"ContainerDied","Data":"873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40"} Nov 03 14:41:29.406018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.405452 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z" event={"ID":"e5304dea-ff1c-4421-8116-2ac7fe91ef73","Type":"ContainerDied","Data":"13ec85fe8676eb15246424e7c939511a4d9d88d80c73ccd6ae0c14edcd988712"} Nov 03 14:41:29.406018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.405476 6389 scope.go:122] "RemoveContainer" containerID="873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40" Nov 03 14:41:29.414537 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.414516 6389 scope.go:122] "RemoveContainer" containerID="873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40" Nov 03 14:41:29.414814 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:41:29.414796 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40\": container with ID starting with 873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40 not found: ID does not exist" containerID="873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40" Nov 03 14:41:29.414896 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.414822 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40"} err="failed to get container status \"873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40\": rpc error: code = NotFound desc = could not find container \"873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40\": container with ID starting with 873c581c8c35b14ae978f28ec10ff848c1376e3935103918b9d74a31e531de40 not found: ID does not exist" Nov 03 14:41:29.460426 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.460394 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z"] Nov 03 14:41:29.472938 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:29.472908 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-554695k5z"] Nov 03 14:41:30.616546 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:30.616511 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5304dea-ff1c-4421-8116-2ac7fe91ef73" path="/var/lib/kubelet/pods/e5304dea-ff1c-4421-8116-2ac7fe91ef73/volumes" Nov 03 14:41:35.076320 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.076286 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28"] Nov 03 14:41:35.076855 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.076837 6389 cpu_manager.go:414] "containerMap: removing container" podUID="be6d1f9b-4748-4a26-bd59-89a17586eeaf" containerName="payload-processing" Nov 03 14:41:35.076855 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.076856 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6d1f9b-4748-4a26-bd59-89a17586eeaf" containerName="payload-processing" Nov 03 14:41:35.076966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.076867 6389 cpu_manager.go:414] "containerMap: removing container" podUID="e5304dea-ff1c-4421-8116-2ac7fe91ef73" containerName="payload-processing" Nov 03 14:41:35.076966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.076872 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5304dea-ff1c-4421-8116-2ac7fe91ef73" containerName="payload-processing" Nov 03 14:41:35.076966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.076960 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6d1f9b-4748-4a26-bd59-89a17586eeaf" containerName="payload-processing" Nov 03 14:41:35.077065 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.076969 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5304dea-ff1c-4421-8116-2ac7fe91ef73" containerName="payload-processing" Nov 03 14:41:35.081416 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.081395 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.099003 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.098975 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.099117 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.099031 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n8bp\" (UniqueName: \"kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-kube-api-access-5n8bp\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.099170 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.099137 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.099210 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.099190 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.102259 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.102232 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-e2e--29c79c46-dockercfg-bqq2f\"" Nov 03 14:41:35.102357 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.102280 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-78c77d81649c\"" Nov 03 14:41:35.104323 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.104302 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28"] Nov 03 14:41:35.114247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.114213 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"openshift-service-ca.crt\"" Nov 03 14:41:35.114247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.114232 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"routing-overlay\"" Nov 03 14:41:35.114247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.114243 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"kube-root-ca.crt\"" Nov 03 14:41:35.114432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.114231 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b4361f\"" Nov 03 14:41:35.200341 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.200303 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.200509 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.200351 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n8bp\" (UniqueName: \"kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-kube-api-access-5n8bp\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.200509 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.200376 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.200509 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.200409 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.201006 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.200982 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.201249 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.201197 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.203013 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.202992 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.222160 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.222135 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n8bp\" (UniqueName: \"kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-kube-api-access-5n8bp\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.394846 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.394749 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:35.701825 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.701800 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28"] Nov 03 14:41:35.703402 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:41:35.703373 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc6d461_cb71_4087_a8ee_254c28d49a31.slice/crio-639ceac1fa6ac5468e575798d5d8a4267f828290fffe5ba5b321e0a37797b341 WatchSource:0}: Error finding container 639ceac1fa6ac5468e575798d5d8a4267f828290fffe5ba5b321e0a37797b341: Status 404 returned error can't find the container with id 639ceac1fa6ac5468e575798d5d8a4267f828290fffe5ba5b321e0a37797b341 Nov 03 14:41:35.950446 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.950368 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p"] Nov 03 14:41:35.954029 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.954009 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:35.976632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.976517 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-5b62df50d3b5\"" Nov 03 14:41:35.977132 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.976843 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-e2e--e9e6f325-dockercfg-nzhch\"" Nov 03 14:41:35.977132 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.976949 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"routing-overlay\"" Nov 03 14:41:35.977132 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.977086 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"kube-root-ca.crt\"" Nov 03 14:41:35.978219 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.978196 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p"] Nov 03 14:41:35.984749 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.984705 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-331c9e\"" Nov 03 14:41:35.984868 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:35.984828 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"openshift-service-ca.crt\"" Nov 03 14:41:36.022279 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.022228 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.022472 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.022277 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.022539 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.022495 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.022724 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.022701 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtk6z\" (UniqueName: \"kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-kube-api-access-jtk6z\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.124201 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.124167 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.124201 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.124204 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.124762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.124274 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.124762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.124336 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtk6z\" (UniqueName: \"kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-kube-api-access-jtk6z\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.125026 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.125001 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.125294 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.125261 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.127018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.126989 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.146357 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.146323 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtk6z\" (UniqueName: \"kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-kube-api-access-jtk6z\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.271918 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.271829 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:36.443460 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.443421 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" event={"ID":"bbc6d461-cb71-4087-a8ee-254c28d49a31","Type":"ContainerStarted","Data":"c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605"} Nov 03 14:41:36.443660 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.443489 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" event={"ID":"bbc6d461-cb71-4087-a8ee-254c28d49a31","Type":"ContainerStarted","Data":"639ceac1fa6ac5468e575798d5d8a4267f828290fffe5ba5b321e0a37797b341"} Nov 03 14:41:36.443660 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.443516 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:36.487382 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.487327 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" podStartSLOduration=1.487314255 podStartE2EDuration="1.487314255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:41:36.486054758 +0000 UTC m=+1772.044691254" watchObservedRunningTime="2026-11-03 14:41:36.487314255 +0000 UTC m=+1772.045950738" Nov 03 14:41:36.636529 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:36.636147 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p"] Nov 03 14:41:37.449982 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:37.449950 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" event={"ID":"be1913b1-3034-4dec-a159-fbc6abbb7276","Type":"ContainerStarted","Data":"2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e"} Nov 03 14:41:37.450405 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:37.450009 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" event={"ID":"be1913b1-3034-4dec-a159-fbc6abbb7276","Type":"ContainerStarted","Data":"4892ccdac846566daaecd8e707905c7ce6c8dc5be0486e401a731bf6b88b554b"} Nov 03 14:41:37.450405 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:37.450191 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:37.493273 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:37.493223 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" podStartSLOduration=2.493208835 podStartE2EDuration="2.493208835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:41:37.492003295 +0000 UTC m=+1773.050639780" watchObservedRunningTime="2026-11-03 14:41:37.493208835 +0000 UTC m=+1773.051845315" Nov 03 14:41:47.453415 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:47.453387 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:41:48.458556 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:41:48.458528 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:41:58.374137 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:41:58.374100 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:42:17.271319 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.271286 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd"] Nov 03 14:42:17.276243 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.276222 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.292973 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.292953 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-3e061afe7424\"" Nov 03 14:42:17.307736 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.307697 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd"] Nov 03 14:42:17.352453 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.352423 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.352603 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.352467 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.352603 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.352563 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.352720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.352702 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlqlq\" (UniqueName: \"kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-kube-api-access-dlqlq\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.453273 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.453241 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.453446 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.453289 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.453446 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.453330 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.453446 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.453397 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlqlq\" (UniqueName: \"kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-kube-api-access-dlqlq\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.454030 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.453966 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.454299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.454072 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.455943 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.455923 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.477101 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.477075 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlqlq\" (UniqueName: \"kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-kube-api-access-dlqlq\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.590251 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.590167 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:17.891048 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:17.891016 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd"] Nov 03 14:42:17.893139 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:42:17.893112 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd0c9f7_6d90_4639_9081_109616e78bc6.slice/crio-b9db2519cdb934125064ba62ffb903ce5837a25ce682579952578837e2099fa7 WatchSource:0}: Error finding container b9db2519cdb934125064ba62ffb903ce5837a25ce682579952578837e2099fa7: Status 404 returned error can't find the container with id b9db2519cdb934125064ba62ffb903ce5837a25ce682579952578837e2099fa7 Nov 03 14:42:18.132505 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.132472 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6"] Nov 03 14:42:18.136201 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.136182 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.155002 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.154975 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-2720f8e2922f\"" Nov 03 14:42:18.156519 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.156488 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6"] Nov 03 14:42:18.161677 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.161650 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.161810 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.161792 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw44x\" (UniqueName: \"kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-kube-api-access-fw44x\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.161859 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.161841 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.161910 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.161876 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.262875 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.262836 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw44x\" (UniqueName: \"kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-kube-api-access-fw44x\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.263084 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.262900 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.263084 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.262927 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.263084 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.262987 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.263550 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.263525 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.263767 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.263743 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.265656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.265635 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.286234 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.286205 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw44x\" (UniqueName: \"kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-kube-api-access-fw44x\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.449437 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.449397 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:18.684936 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.684897 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" event={"ID":"afd0c9f7-6d90-4639-9081-109616e78bc6","Type":"ContainerStarted","Data":"3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841"} Nov 03 14:42:18.685102 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.684952 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" event={"ID":"afd0c9f7-6d90-4639-9081-109616e78bc6","Type":"ContainerStarted","Data":"b9db2519cdb934125064ba62ffb903ce5837a25ce682579952578837e2099fa7"} Nov 03 14:42:18.685102 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.685020 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:18.748915 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.748799 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" podStartSLOduration=1.748783016 podStartE2EDuration="1.748783016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:42:18.748454245 +0000 UTC m=+1814.307090730" watchObservedRunningTime="2026-11-03 14:42:18.748783016 +0000 UTC m=+1814.307419500" Nov 03 14:42:18.763515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:18.763483 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6"] Nov 03 14:42:19.690772 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:19.690735 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" event={"ID":"9a646870-36d1-4d20-b25b-59667aecd959","Type":"ContainerStarted","Data":"16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f"} Nov 03 14:42:19.691234 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:19.690798 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" event={"ID":"9a646870-36d1-4d20-b25b-59667aecd959","Type":"ContainerStarted","Data":"e90ff494509feb24324964061e805dbed92dc9628a57308046e49790bb48d9af"} Nov 03 14:42:19.691234 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:19.690979 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:19.735474 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:19.735420 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" podStartSLOduration=1.7354055769999999 podStartE2EDuration="1.735405577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:42:19.734185739 +0000 UTC m=+1815.292822224" watchObservedRunningTime="2026-11-03 14:42:19.735405577 +0000 UTC m=+1815.294042095" Nov 03 14:42:28.374370 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:42:28.374333 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:42:29.694801 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:29.694772 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:42:29.771680 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:29.771632 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28"] Nov 03 14:42:30.698527 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:30.698496 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:42:30.778215 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:30.778179 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p"] Nov 03 14:42:34.772368 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:34.772329 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" podUID="bbc6d461-cb71-4087-a8ee-254c28d49a31" containerName="payload-processing" containerID="cri-o://c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605" gracePeriod=25 Nov 03 14:42:35.025259 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.025190 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:42:35.160110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.160083 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-kube-api-access-5n8bp\" (UniqueName: \"kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-kube-api-access-5n8bp\") pod \"bbc6d461-cb71-4087-a8ee-254c28d49a31\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " Nov 03 14:42:35.160312 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.160182 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-routing-overlay-volume\") pod \"bbc6d461-cb71-4087-a8ee-254c28d49a31\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " Nov 03 14:42:35.160312 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.160210 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-plugins-config-volume\") pod \"bbc6d461-cb71-4087-a8ee-254c28d49a31\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " Nov 03 14:42:35.160312 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.160236 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-provider-credentials\") pod \"bbc6d461-cb71-4087-a8ee-254c28d49a31\" (UID: \"bbc6d461-cb71-4087-a8ee-254c28d49a31\") " Nov 03 14:42:35.160654 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.160593 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-routing-overlay-volume" pod "bbc6d461-cb71-4087-a8ee-254c28d49a31" (UID: "bbc6d461-cb71-4087-a8ee-254c28d49a31"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:42:35.160728 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.160647 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-plugins-config-volume" pod "bbc6d461-cb71-4087-a8ee-254c28d49a31" (UID: "bbc6d461-cb71-4087-a8ee-254c28d49a31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:42:35.160772 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.160764 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:42:35.160806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.160777 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6d461-cb71-4087-a8ee-254c28d49a31-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:42:35.162314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.162288 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-kube-api-access-5n8bp" pod "bbc6d461-cb71-4087-a8ee-254c28d49a31" (UID: "bbc6d461-cb71-4087-a8ee-254c28d49a31"). InnerVolumeSpecName "kube-api-access-5n8bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:42:35.162441 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.162370 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-provider-credentials" pod "bbc6d461-cb71-4087-a8ee-254c28d49a31" (UID: "bbc6d461-cb71-4087-a8ee-254c28d49a31"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:42:35.261995 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.261965 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:42:35.261995 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.261991 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n8bp\" (UniqueName: \"kubernetes.io/projected/bbc6d461-cb71-4087-a8ee-254c28d49a31-kube-api-access-5n8bp\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:42:35.771110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.771071 6389 generic.go:356] "Generic (PLEG): container finished" podID="bbc6d461-cb71-4087-a8ee-254c28d49a31" containerID="c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605" exitCode=0 Nov 03 14:42:35.771303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.771135 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" Nov 03 14:42:35.771303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.771142 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" event={"ID":"bbc6d461-cb71-4087-a8ee-254c28d49a31","Type":"ContainerDied","Data":"c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605"} Nov 03 14:42:35.771303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.771191 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28" event={"ID":"bbc6d461-cb71-4087-a8ee-254c28d49a31","Type":"ContainerDied","Data":"639ceac1fa6ac5468e575798d5d8a4267f828290fffe5ba5b321e0a37797b341"} Nov 03 14:42:35.771303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.771212 6389 scope.go:122] "RemoveContainer" containerID="c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605" Nov 03 14:42:35.778511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.778452 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" podUID="be1913b1-3034-4dec-a159-fbc6abbb7276" containerName="payload-processing" containerID="cri-o://2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e" gracePeriod=25 Nov 03 14:42:35.780819 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.780801 6389 scope.go:122] "RemoveContainer" containerID="c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605" Nov 03 14:42:35.781090 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:42:35.781062 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605\": container with ID starting with c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605 not found: ID does not exist" containerID="c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605" Nov 03 14:42:35.781183 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.781100 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605"} err="failed to get container status \"c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605\": rpc error: code = NotFound desc = could not find container \"c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605\": container with ID starting with c6e375bc888843189dd505c27f05c57129dc9b949245d0f0ef98478d6dab4605 not found: ID does not exist" Nov 03 14:42:35.828238 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.828200 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28"] Nov 03 14:42:35.839959 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:35.839934 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b87976w28"] Nov 03 14:42:36.035223 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.035159 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:42:36.170067 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.170028 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-routing-overlay-volume\") pod \"be1913b1-3034-4dec-a159-fbc6abbb7276\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " Nov 03 14:42:36.170250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.170167 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-provider-credentials\" (UniqueName: \"kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-provider-credentials\") pod \"be1913b1-3034-4dec-a159-fbc6abbb7276\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " Nov 03 14:42:36.170250 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.170196 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-plugins-config-volume\") pod \"be1913b1-3034-4dec-a159-fbc6abbb7276\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " Nov 03 14:42:36.170369 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.170317 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-kube-api-access-jtk6z\" (UniqueName: \"kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-kube-api-access-jtk6z\") pod \"be1913b1-3034-4dec-a159-fbc6abbb7276\" (UID: \"be1913b1-3034-4dec-a159-fbc6abbb7276\") " Nov 03 14:42:36.170369 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.170349 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-routing-overlay-volume" pod "be1913b1-3034-4dec-a159-fbc6abbb7276" (UID: "be1913b1-3034-4dec-a159-fbc6abbb7276"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:42:36.170662 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.170645 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:42:36.170747 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.170668 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-plugins-config-volume" pod "be1913b1-3034-4dec-a159-fbc6abbb7276" (UID: "be1913b1-3034-4dec-a159-fbc6abbb7276"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:42:36.172450 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.172430 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-provider-credentials" pod "be1913b1-3034-4dec-a159-fbc6abbb7276" (UID: "be1913b1-3034-4dec-a159-fbc6abbb7276"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:42:36.172544 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.172451 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-kube-api-access-jtk6z" pod "be1913b1-3034-4dec-a159-fbc6abbb7276" (UID: "be1913b1-3034-4dec-a159-fbc6abbb7276"). InnerVolumeSpecName "kube-api-access-jtk6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:42:36.271904 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.271872 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:42:36.271904 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.271897 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1913b1-3034-4dec-a159-fbc6abbb7276-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:42:36.271904 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.271908 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtk6z\" (UniqueName: \"kubernetes.io/projected/be1913b1-3034-4dec-a159-fbc6abbb7276-kube-api-access-jtk6z\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:42:36.616919 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.616891 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc6d461-cb71-4087-a8ee-254c28d49a31" path="/var/lib/kubelet/pods/bbc6d461-cb71-4087-a8ee-254c28d49a31/volumes" Nov 03 14:42:36.777721 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.777689 6389 generic.go:356] "Generic (PLEG): container finished" podID="be1913b1-3034-4dec-a159-fbc6abbb7276" containerID="2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e" exitCode=0 Nov 03 14:42:36.777907 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.777732 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" Nov 03 14:42:36.777907 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.777777 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" event={"ID":"be1913b1-3034-4dec-a159-fbc6abbb7276","Type":"ContainerDied","Data":"2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e"} Nov 03 14:42:36.777907 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.777827 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p" event={"ID":"be1913b1-3034-4dec-a159-fbc6abbb7276","Type":"ContainerDied","Data":"4892ccdac846566daaecd8e707905c7ce6c8dc5be0486e401a731bf6b88b554b"} Nov 03 14:42:36.777907 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.777847 6389 scope.go:122] "RemoveContainer" containerID="2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e" Nov 03 14:42:36.786505 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.786331 6389 scope.go:122] "RemoveContainer" containerID="2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e" Nov 03 14:42:36.786772 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:42:36.786586 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e\": container with ID starting with 2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e not found: ID does not exist" containerID="2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e" Nov 03 14:42:36.786772 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.786626 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e"} err="failed to get container status \"2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e\": rpc error: code = NotFound desc = could not find container \"2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e\": container with ID starting with 2473463d951d47aec102f5800a50b355dfb9857b620faed6ab18130916860a2e not found: ID does not exist" Nov 03 14:42:36.834565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.834538 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p"] Nov 03 14:42:36.847025 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:36.847003 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758frts4p"] Nov 03 14:42:38.616999 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:38.616971 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1913b1-3034-4dec-a159-fbc6abbb7276" path="/var/lib/kubelet/pods/be1913b1-3034-4dec-a159-fbc6abbb7276/volumes" Nov 03 14:42:58.400367 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:42:58.400327 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:42:59.176668 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.176635 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s"] Nov 03 14:42:59.177092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.177076 6389 cpu_manager.go:414] "containerMap: removing container" podUID="be1913b1-3034-4dec-a159-fbc6abbb7276" containerName="payload-processing" Nov 03 14:42:59.177092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.177091 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1913b1-3034-4dec-a159-fbc6abbb7276" containerName="payload-processing" Nov 03 14:42:59.177259 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.177119 6389 cpu_manager.go:414] "containerMap: removing container" podUID="bbc6d461-cb71-4087-a8ee-254c28d49a31" containerName="payload-processing" Nov 03 14:42:59.177259 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.177124 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc6d461-cb71-4087-a8ee-254c28d49a31" containerName="payload-processing" Nov 03 14:42:59.177259 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.177206 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc6d461-cb71-4087-a8ee-254c28d49a31" containerName="payload-processing" Nov 03 14:42:59.177259 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.177221 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1913b1-3034-4dec-a159-fbc6abbb7276" containerName="payload-processing" Nov 03 14:42:59.179846 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.179828 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.211100 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.211069 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-3f3c5a621d3c\"" Nov 03 14:42:59.212825 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.212796 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s"] Nov 03 14:42:59.308530 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.308488 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.308739 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.308669 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58224\" (UniqueName: \"kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-kube-api-access-58224\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.308739 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.308720 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.308821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.308780 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.409593 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.409556 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58224\" (UniqueName: \"kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-kube-api-access-58224\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.410086 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.409637 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.410086 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.409666 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.410086 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.409716 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.410464 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.410440 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.410548 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.410467 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.412483 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.412457 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.433122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.433066 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58224\" (UniqueName: \"kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-kube-api-access-58224\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.493355 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.493322 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.795999 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.795973 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s"] Nov 03 14:42:59.797810 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:42:59.797780 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e51ad9_7c5b_4e9f_88f5_d29e2353bacd.slice/crio-f032accb0059e8db96c872bf3fa1fc6e1e313bd425642e9a44009cd4d7f9f8ff WatchSource:0}: Error finding container f032accb0059e8db96c872bf3fa1fc6e1e313bd425642e9a44009cd4d7f9f8ff: Status 404 returned error can't find the container with id f032accb0059e8db96c872bf3fa1fc6e1e313bd425642e9a44009cd4d7f9f8ff Nov 03 14:42:59.894843 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.894811 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" event={"ID":"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd","Type":"ContainerStarted","Data":"a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370"} Nov 03 14:42:59.895010 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.894864 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" event={"ID":"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd","Type":"ContainerStarted","Data":"f032accb0059e8db96c872bf3fa1fc6e1e313bd425642e9a44009cd4d7f9f8ff"} Nov 03 14:42:59.895010 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.894887 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:42:59.941924 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:42:59.941869 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" podStartSLOduration=0.941854995 podStartE2EDuration="941.854995ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:42:59.941354596 +0000 UTC m=+1855.499991080" watchObservedRunningTime="2026-11-03 14:42:59.941854995 +0000 UTC m=+1855.500491478" Nov 03 14:43:00.040760 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.040730 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd"] Nov 03 14:43:00.044381 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.044364 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.068005 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.067979 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd"] Nov 03 14:43:00.071649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.071596 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-084a61fd1674\"" Nov 03 14:43:00.118676 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.118638 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xvc\" (UniqueName: \"kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-kube-api-access-m7xvc\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.118834 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.118693 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.118895 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.118856 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.118895 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.118889 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.219680 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.219638 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xvc\" (UniqueName: \"kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-kube-api-access-m7xvc\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.219891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.219698 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.219891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.219812 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.219891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.219841 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.220378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.220351 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.220561 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.220537 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.222562 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.222540 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.242385 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.242356 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xvc\" (UniqueName: \"kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-kube-api-access-m7xvc\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.360227 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.360142 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:00.676748 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.676716 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd"] Nov 03 14:43:00.694079 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:43:00.694045 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ccbfa7_1a63_4e96_a74a_5888ceb705b3.slice/crio-9b31864cc7db6986090d455d19b9092b48cb397d0cb6401e4e776f46af3877f0 WatchSource:0}: Error finding container 9b31864cc7db6986090d455d19b9092b48cb397d0cb6401e4e776f46af3877f0: Status 404 returned error can't find the container with id 9b31864cc7db6986090d455d19b9092b48cb397d0cb6401e4e776f46af3877f0 Nov 03 14:43:00.914511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.911032 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" event={"ID":"43ccbfa7-1a63-4e96-a74a-5888ceb705b3","Type":"ContainerStarted","Data":"71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc"} Nov 03 14:43:00.914511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.911129 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" event={"ID":"43ccbfa7-1a63-4e96-a74a-5888ceb705b3","Type":"ContainerStarted","Data":"9b31864cc7db6986090d455d19b9092b48cb397d0cb6401e4e776f46af3877f0"} Nov 03 14:43:00.914511 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:00.911194 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:04.975697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:04.975640 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" podStartSLOduration=4.9756228799999995 podStartE2EDuration="4.97562288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:43:00.965039004 +0000 UTC m=+1856.523675488" watchObservedRunningTime="2026-11-03 14:43:04.97562288 +0000 UTC m=+1860.534259363" Nov 03 14:43:04.976783 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:04.976766 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8"] Nov 03 14:43:04.981727 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:04.981697 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" Nov 03 14:43:05.005321 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.005290 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8"] Nov 03 14:43:05.016037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.016007 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-96714c\"/\"openshift-service-ca.crt\"" Nov 03 14:43:05.016037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.016011 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-96714c\"/\"kube-root-ca.crt\"" Nov 03 14:43:05.016237 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.016015 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-96714c\"/\"default-dockercfg-4ddl8\"" Nov 03 14:43:05.016237 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.016016 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-96714c\"/\"e2e-usage-sim-6293f4-cert\"" Nov 03 14:43:05.089419 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.089388 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4966\" (UniqueName: \"kubernetes.io/projected/625c5998-e17c-4a6e-b885-84d9bd5b41a6-kube-api-access-m4966\") pod \"e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8\" (UID: \"625c5998-e17c-4a6e-b885-84d9bd5b41a6\") " pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" Nov 03 14:43:05.089646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.089546 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/625c5998-e17c-4a6e-b885-84d9bd5b41a6-tls\") pod \"e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8\" (UID: \"625c5998-e17c-4a6e-b885-84d9bd5b41a6\") " pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" Nov 03 14:43:05.191238 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.191205 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/625c5998-e17c-4a6e-b885-84d9bd5b41a6-tls\") pod \"e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8\" (UID: \"625c5998-e17c-4a6e-b885-84d9bd5b41a6\") " pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" Nov 03 14:43:05.191422 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.191285 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4966\" (UniqueName: \"kubernetes.io/projected/625c5998-e17c-4a6e-b885-84d9bd5b41a6-kube-api-access-m4966\") pod \"e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8\" (UID: \"625c5998-e17c-4a6e-b885-84d9bd5b41a6\") " pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" Nov 03 14:43:05.193838 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.193812 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/625c5998-e17c-4a6e-b885-84d9bd5b41a6-tls\") pod \"e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8\" (UID: \"625c5998-e17c-4a6e-b885-84d9bd5b41a6\") " pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" Nov 03 14:43:05.216434 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.216406 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4966\" (UniqueName: \"kubernetes.io/projected/625c5998-e17c-4a6e-b885-84d9bd5b41a6-kube-api-access-m4966\") pod \"e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8\" (UID: \"625c5998-e17c-4a6e-b885-84d9bd5b41a6\") " pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" Nov 03 14:43:05.293710 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.293583 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" Nov 03 14:43:05.599921 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.599896 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8"] Nov 03 14:43:05.601798 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:43:05.601771 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod625c5998_e17c_4a6e_b885_84d9bd5b41a6.slice/crio-eeb23d3fc0bf82d706b962149ed0470385031ee2901ec2d57015954cd5ed365b WatchSource:0}: Error finding container eeb23d3fc0bf82d706b962149ed0470385031ee2901ec2d57015954cd5ed365b: Status 404 returned error can't find the container with id eeb23d3fc0bf82d706b962149ed0470385031ee2901ec2d57015954cd5ed365b Nov 03 14:43:05.936519 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.936483 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" event={"ID":"625c5998-e17c-4a6e-b885-84d9bd5b41a6","Type":"ContainerStarted","Data":"87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7"} Nov 03 14:43:05.936519 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.936524 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" event={"ID":"625c5998-e17c-4a6e-b885-84d9bd5b41a6","Type":"ContainerStarted","Data":"eeb23d3fc0bf82d706b962149ed0470385031ee2901ec2d57015954cd5ed365b"} Nov 03 14:43:05.936815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.936718 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" Nov 03 14:43:05.941478 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.941458 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" Nov 03 14:43:05.980402 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:05.980346 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" podStartSLOduration=1.98033312 podStartE2EDuration="1.98033312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:43:05.977993345 +0000 UTC m=+1861.536629830" watchObservedRunningTime="2026-11-03 14:43:05.98033312 +0000 UTC m=+1861.538969603" Nov 03 14:43:10.913492 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:10.913455 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:43:10.993564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:10.993534 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd"] Nov 03 14:43:12.450442 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.450379 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd"] Nov 03 14:43:12.462329 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.462299 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.476917 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.476891 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd"] Nov 03 14:43:12.487102 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.487077 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-404dd2fdb382\"" Nov 03 14:43:12.578654 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.578596 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vrp9\" (UniqueName: \"kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-kube-api-access-6vrp9\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.578861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.578706 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.578861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.578856 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.578955 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.578905 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.679716 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.679679 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.679933 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.679744 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.679933 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.679787 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vrp9\" (UniqueName: \"kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-kube-api-access-6vrp9\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.679933 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.679819 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.680407 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.680378 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.680407 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.680400 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.682870 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.682842 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.708042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.707968 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vrp9\" (UniqueName: \"kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-kube-api-access-6vrp9\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.778416 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.778375 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:12.922973 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:12.922945 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:13.002096 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:13.001998 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6"] Nov 03 14:43:13.087946 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:13.087922 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd"] Nov 03 14:43:13.089908 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:43:13.089881 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24c7af1_4826_4321_a38a_36a980d31b68.slice/crio-a777f460d02c5ebcc6f60e9a89caba14fbb50eb09abe7fd5360236e357882e32 WatchSource:0}: Error finding container a777f460d02c5ebcc6f60e9a89caba14fbb50eb09abe7fd5360236e357882e32: Status 404 returned error can't find the container with id a777f460d02c5ebcc6f60e9a89caba14fbb50eb09abe7fd5360236e357882e32 Nov 03 14:43:13.978810 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:13.978768 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" event={"ID":"b24c7af1-4826-4321-a38a-36a980d31b68","Type":"ContainerStarted","Data":"465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce"} Nov 03 14:43:13.979246 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:13.978831 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" event={"ID":"b24c7af1-4826-4321-a38a-36a980d31b68","Type":"ContainerStarted","Data":"a777f460d02c5ebcc6f60e9a89caba14fbb50eb09abe7fd5360236e357882e32"} Nov 03 14:43:13.979246 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:13.978956 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:14.022423 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:14.022373 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" podStartSLOduration=2.022356751 podStartE2EDuration="2.022356751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:43:14.021221565 +0000 UTC m=+1869.579858049" watchObservedRunningTime="2026-11-03 14:43:14.022356751 +0000 UTC m=+1869.580993235" Nov 03 14:43:15.993921 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:15.993885 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" podUID="afd0c9f7-6d90-4639-9081-109616e78bc6" containerName="payload-processing" containerID="cri-o://3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841" gracePeriod=25 Nov 03 14:43:16.239832 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.239809 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:43:16.317049 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.316972 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-plugins-config-volume\") pod \"afd0c9f7-6d90-4639-9081-109616e78bc6\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " Nov 03 14:43:16.317049 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.317034 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-kube-api-access-dlqlq\" (UniqueName: \"kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-kube-api-access-dlqlq\") pod \"afd0c9f7-6d90-4639-9081-109616e78bc6\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " Nov 03 14:43:16.317283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.317098 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-provider-credentials\") pod \"afd0c9f7-6d90-4639-9081-109616e78bc6\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " Nov 03 14:43:16.317283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.317235 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-routing-overlay-volume\") pod \"afd0c9f7-6d90-4639-9081-109616e78bc6\" (UID: \"afd0c9f7-6d90-4639-9081-109616e78bc6\") " Nov 03 14:43:16.317446 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.317419 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-plugins-config-volume" pod "afd0c9f7-6d90-4639-9081-109616e78bc6" (UID: "afd0c9f7-6d90-4639-9081-109616e78bc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:43:16.317564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.317547 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:16.317813 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.317785 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-routing-overlay-volume" pod "afd0c9f7-6d90-4639-9081-109616e78bc6" (UID: "afd0c9f7-6d90-4639-9081-109616e78bc6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:43:16.320119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.320089 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-kube-api-access-dlqlq" pod "afd0c9f7-6d90-4639-9081-109616e78bc6" (UID: "afd0c9f7-6d90-4639-9081-109616e78bc6"). InnerVolumeSpecName "kube-api-access-dlqlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:43:16.320244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.320183 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-provider-credentials" pod "afd0c9f7-6d90-4639-9081-109616e78bc6" (UID: "afd0c9f7-6d90-4639-9081-109616e78bc6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:43:16.418160 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.418126 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlqlq\" (UniqueName: \"kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-kube-api-access-dlqlq\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:16.418160 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.418155 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/afd0c9f7-6d90-4639-9081-109616e78bc6-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:16.418160 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.418165 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/afd0c9f7-6d90-4639-9081-109616e78bc6-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:16.993840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.993803 6389 generic.go:356] "Generic (PLEG): container finished" podID="afd0c9f7-6d90-4639-9081-109616e78bc6" containerID="3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841" exitCode=0 Nov 03 14:43:16.994042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.993851 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" Nov 03 14:43:16.994042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.993879 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" event={"ID":"afd0c9f7-6d90-4639-9081-109616e78bc6","Type":"ContainerDied","Data":"3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841"} Nov 03 14:43:16.994042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.993934 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd" event={"ID":"afd0c9f7-6d90-4639-9081-109616e78bc6","Type":"ContainerDied","Data":"b9db2519cdb934125064ba62ffb903ce5837a25ce682579952578837e2099fa7"} Nov 03 14:43:16.994042 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:16.993955 6389 scope.go:122] "RemoveContainer" containerID="3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841" Nov 03 14:43:17.002656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:17.002637 6389 scope.go:122] "RemoveContainer" containerID="3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841" Nov 03 14:43:17.002903 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:43:17.002885 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841\": container with ID starting with 3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841 not found: ID does not exist" containerID="3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841" Nov 03 14:43:17.002960 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:17.002913 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841"} err="failed to get container status \"3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841\": rpc error: code = NotFound desc = could not find container \"3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841\": container with ID starting with 3d48a8fece840a82a959c9a5648e470bdf426c1fe4d2928c60edea0e292f0841 not found: ID does not exist" Nov 03 14:43:17.052984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:17.052959 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd"] Nov 03 14:43:17.065755 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:17.065724 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4tjsqd"] Nov 03 14:43:18.002626 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.002569 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" podUID="9a646870-36d1-4d20-b25b-59667aecd959" containerName="payload-processing" containerID="cri-o://16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f" gracePeriod=25 Nov 03 14:43:18.262739 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.262681 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:43:18.339496 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.339467 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-kube-api-access-fw44x\" (UniqueName: \"kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-kube-api-access-fw44x\") pod \"9a646870-36d1-4d20-b25b-59667aecd959\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " Nov 03 14:43:18.339720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.339576 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-plugins-config-volume\") pod \"9a646870-36d1-4d20-b25b-59667aecd959\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " Nov 03 14:43:18.339720 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.339649 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-routing-overlay-volume\") pod \"9a646870-36d1-4d20-b25b-59667aecd959\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " Nov 03 14:43:18.339862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.339775 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-provider-credentials\") pod \"9a646870-36d1-4d20-b25b-59667aecd959\" (UID: \"9a646870-36d1-4d20-b25b-59667aecd959\") " Nov 03 14:43:18.340253 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.339999 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-plugins-config-volume" pod "9a646870-36d1-4d20-b25b-59667aecd959" (UID: "9a646870-36d1-4d20-b25b-59667aecd959"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:43:18.340253 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.340052 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-routing-overlay-volume" pod "9a646870-36d1-4d20-b25b-59667aecd959" (UID: "9a646870-36d1-4d20-b25b-59667aecd959"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:43:18.340253 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.340188 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:18.340253 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.340210 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a646870-36d1-4d20-b25b-59667aecd959-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:18.342208 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.342177 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-provider-credentials" pod "9a646870-36d1-4d20-b25b-59667aecd959" (UID: "9a646870-36d1-4d20-b25b-59667aecd959"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:43:18.342874 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.342854 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-kube-api-access-fw44x" pod "9a646870-36d1-4d20-b25b-59667aecd959" (UID: "9a646870-36d1-4d20-b25b-59667aecd959"). InnerVolumeSpecName "kube-api-access-fw44x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:43:18.441158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.441127 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:18.441158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.441154 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw44x\" (UniqueName: \"kubernetes.io/projected/9a646870-36d1-4d20-b25b-59667aecd959-kube-api-access-fw44x\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:18.617286 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:18.617210 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd0c9f7-6d90-4639-9081-109616e78bc6" path="/var/lib/kubelet/pods/afd0c9f7-6d90-4639-9081-109616e78bc6/volumes" Nov 03 14:43:19.006180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:19.006149 6389 generic.go:356] "Generic (PLEG): container finished" podID="9a646870-36d1-4d20-b25b-59667aecd959" containerID="16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f" exitCode=0 Nov 03 14:43:19.006659 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:19.006194 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" Nov 03 14:43:19.006659 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:19.006237 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" event={"ID":"9a646870-36d1-4d20-b25b-59667aecd959","Type":"ContainerDied","Data":"16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f"} Nov 03 14:43:19.006659 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:19.006293 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6" event={"ID":"9a646870-36d1-4d20-b25b-59667aecd959","Type":"ContainerDied","Data":"e90ff494509feb24324964061e805dbed92dc9628a57308046e49790bb48d9af"} Nov 03 14:43:19.006659 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:19.006321 6389 scope.go:122] "RemoveContainer" containerID="16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f" Nov 03 14:43:19.015230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:19.015210 6389 scope.go:122] "RemoveContainer" containerID="16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f" Nov 03 14:43:19.015508 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:43:19.015489 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f\": container with ID starting with 16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f not found: ID does not exist" containerID="16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f" Nov 03 14:43:19.015557 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:19.015515 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f"} err="failed to get container status \"16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f\": rpc error: code = NotFound desc = could not find container \"16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f\": container with ID starting with 16a5ef526fe528396cd18bea34b766b208cdc8eca7a3d22b9d153f9946ecf82f not found: ID does not exist" Nov 03 14:43:19.061923 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:19.061894 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6"] Nov 03 14:43:19.074470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:19.074446 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865s6lh6"] Nov 03 14:43:20.618515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:20.618482 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a646870-36d1-4d20-b25b-59667aecd959" path="/var/lib/kubelet/pods/9a646870-36d1-4d20-b25b-59667aecd959/volumes" Nov 03 14:43:24.986774 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:24.986744 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:43:25.064226 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:25.064189 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s"] Nov 03 14:43:26.637130 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:26.637005 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd"] Nov 03 14:43:28.374339 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:43:28.374295 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:43:30.065448 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.065407 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" podUID="16e51ad9-7c5b-4e9f-88f5-d29e2353bacd" containerName="payload-processing" containerID="cri-o://a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370" gracePeriod=25 Nov 03 14:43:30.328299 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.328234 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:43:30.376695 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.376660 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-kube-api-access-58224\" (UniqueName: \"kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-kube-api-access-58224\") pod \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " Nov 03 14:43:30.376873 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.376752 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-plugins-config-volume\") pod \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " Nov 03 14:43:30.376873 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.376817 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-routing-overlay-volume\") pod \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " Nov 03 14:43:30.376873 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.376855 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-provider-credentials\") pod \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\" (UID: \"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd\") " Nov 03 14:43:30.377247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.377213 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-plugins-config-volume" pod "16e51ad9-7c5b-4e9f-88f5-d29e2353bacd" (UID: "16e51ad9-7c5b-4e9f-88f5-d29e2353bacd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:43:30.377247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.377221 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-routing-overlay-volume" pod "16e51ad9-7c5b-4e9f-88f5-d29e2353bacd" (UID: "16e51ad9-7c5b-4e9f-88f5-d29e2353bacd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:43:30.379570 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.379546 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-kube-api-access-58224" pod "16e51ad9-7c5b-4e9f-88f5-d29e2353bacd" (UID: "16e51ad9-7c5b-4e9f-88f5-d29e2353bacd"). InnerVolumeSpecName "kube-api-access-58224". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:43:30.379994 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.379962 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-provider-credentials" pod "16e51ad9-7c5b-4e9f-88f5-d29e2353bacd" (UID: "16e51ad9-7c5b-4e9f-88f5-d29e2353bacd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:43:30.479207 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.479172 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58224\" (UniqueName: \"kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-kube-api-access-58224\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:30.479207 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.479205 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:30.479416 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.479218 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:30.479416 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:30.479230 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:31.070667 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.070601 6389 generic.go:356] "Generic (PLEG): container finished" podID="16e51ad9-7c5b-4e9f-88f5-d29e2353bacd" containerID="a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370" exitCode=0 Nov 03 14:43:31.070667 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.070670 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" Nov 03 14:43:31.071143 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.070713 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" event={"ID":"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd","Type":"ContainerDied","Data":"a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370"} Nov 03 14:43:31.071143 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.070758 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s" event={"ID":"16e51ad9-7c5b-4e9f-88f5-d29e2353bacd","Type":"ContainerDied","Data":"f032accb0059e8db96c872bf3fa1fc6e1e313bd425642e9a44009cd4d7f9f8ff"} Nov 03 14:43:31.071143 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.070780 6389 scope.go:122] "RemoveContainer" containerID="a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370" Nov 03 14:43:31.079486 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.079468 6389 scope.go:122] "RemoveContainer" containerID="a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370" Nov 03 14:43:31.079782 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:43:31.079763 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370\": container with ID starting with a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370 not found: ID does not exist" containerID="a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370" Nov 03 14:43:31.079879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.079789 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370"} err="failed to get container status \"a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370\": rpc error: code = NotFound desc = could not find container \"a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370\": container with ID starting with a76fc0c6db2ffa15bddcafe44ea5bd5a22b44afe45545286ac2aed19e173b370 not found: ID does not exist" Nov 03 14:43:31.126704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.126675 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s"] Nov 03 14:43:31.139406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.139378 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8jzs6s"] Nov 03 14:43:31.638067 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.638024 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" podUID="43ccbfa7-1a63-4e96-a74a-5888ceb705b3" containerName="payload-processing" containerID="cri-o://71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc" gracePeriod=25 Nov 03 14:43:31.900016 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.899958 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:31.997461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.997426 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-kube-api-access-m7xvc\" (UniqueName: \"kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-kube-api-access-m7xvc\") pod \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " Nov 03 14:43:31.997461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.997467 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-plugins-config-volume\") pod \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " Nov 03 14:43:31.997695 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.997649 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-routing-overlay-volume\") pod \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " Nov 03 14:43:31.997695 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.997681 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-provider-credentials\") pod \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\" (UID: \"43ccbfa7-1a63-4e96-a74a-5888ceb705b3\") " Nov 03 14:43:31.998017 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.997991 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-plugins-config-volume" pod "43ccbfa7-1a63-4e96-a74a-5888ceb705b3" (UID: "43ccbfa7-1a63-4e96-a74a-5888ceb705b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:43:31.998017 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.998001 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-routing-overlay-volume" pod "43ccbfa7-1a63-4e96-a74a-5888ceb705b3" (UID: "43ccbfa7-1a63-4e96-a74a-5888ceb705b3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:43:31.998153 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.998110 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:31.998153 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.998127 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:31.999763 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.999746 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-kube-api-access-m7xvc" pod "43ccbfa7-1a63-4e96-a74a-5888ceb705b3" (UID: "43ccbfa7-1a63-4e96-a74a-5888ceb705b3"). InnerVolumeSpecName "kube-api-access-m7xvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:43:31.999921 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:31.999900 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-provider-credentials" pod "43ccbfa7-1a63-4e96-a74a-5888ceb705b3" (UID: "43ccbfa7-1a63-4e96-a74a-5888ceb705b3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:43:32.078019 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.077986 6389 generic.go:356] "Generic (PLEG): container finished" podID="43ccbfa7-1a63-4e96-a74a-5888ceb705b3" containerID="71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc" exitCode=0 Nov 03 14:43:32.078019 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.078023 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" event={"ID":"43ccbfa7-1a63-4e96-a74a-5888ceb705b3","Type":"ContainerDied","Data":"71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc"} Nov 03 14:43:32.078485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.078037 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" Nov 03 14:43:32.078485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.078087 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd" event={"ID":"43ccbfa7-1a63-4e96-a74a-5888ceb705b3","Type":"ContainerDied","Data":"9b31864cc7db6986090d455d19b9092b48cb397d0cb6401e4e776f46af3877f0"} Nov 03 14:43:32.078485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.078110 6389 scope.go:122] "RemoveContainer" containerID="71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc" Nov 03 14:43:32.087227 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.087209 6389 scope.go:122] "RemoveContainer" containerID="71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc" Nov 03 14:43:32.087465 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:43:32.087446 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc\": container with ID starting with 71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc not found: ID does not exist" containerID="71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc" Nov 03 14:43:32.087517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.087475 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc"} err="failed to get container status \"71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc\": rpc error: code = NotFound desc = could not find container \"71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc\": container with ID starting with 71ec1ce7901e1155f151d2492a06962c900e131e76d01cf8e743e38fa031e9fc not found: ID does not exist" Nov 03 14:43:32.098645 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.098603 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:32.098740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.098648 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7xvc\" (UniqueName: \"kubernetes.io/projected/43ccbfa7-1a63-4e96-a74a-5888ceb705b3-kube-api-access-m7xvc\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:43:32.135671 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.135647 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd"] Nov 03 14:43:32.141392 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.141369 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4f66qd"] Nov 03 14:43:32.619089 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.619057 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e51ad9-7c5b-4e9f-88f5-d29e2353bacd" path="/var/lib/kubelet/pods/16e51ad9-7c5b-4e9f-88f5-d29e2353bacd/volumes" Nov 03 14:43:32.619454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:32.619440 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ccbfa7-1a63-4e96-a74a-5888ceb705b3" path="/var/lib/kubelet/pods/43ccbfa7-1a63-4e96-a74a-5888ceb705b3/volumes" Nov 03 14:43:35.190155 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190120 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb"] Nov 03 14:43:35.190631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190528 6389 cpu_manager.go:414] "containerMap: removing container" podUID="afd0c9f7-6d90-4639-9081-109616e78bc6" containerName="payload-processing" Nov 03 14:43:35.190631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190539 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd0c9f7-6d90-4639-9081-109616e78bc6" containerName="payload-processing" Nov 03 14:43:35.190631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190554 6389 cpu_manager.go:414] "containerMap: removing container" podUID="43ccbfa7-1a63-4e96-a74a-5888ceb705b3" containerName="payload-processing" Nov 03 14:43:35.190631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190562 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ccbfa7-1a63-4e96-a74a-5888ceb705b3" containerName="payload-processing" Nov 03 14:43:35.190631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190589 6389 cpu_manager.go:414] "containerMap: removing container" podUID="9a646870-36d1-4d20-b25b-59667aecd959" containerName="payload-processing" Nov 03 14:43:35.190631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190597 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a646870-36d1-4d20-b25b-59667aecd959" containerName="payload-processing" Nov 03 14:43:35.190862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190651 6389 cpu_manager.go:414] "containerMap: removing container" podUID="16e51ad9-7c5b-4e9f-88f5-d29e2353bacd" containerName="payload-processing" Nov 03 14:43:35.190862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190660 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e51ad9-7c5b-4e9f-88f5-d29e2353bacd" containerName="payload-processing" Nov 03 14:43:35.190862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190761 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd0c9f7-6d90-4639-9081-109616e78bc6" containerName="payload-processing" Nov 03 14:43:35.190862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190772 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ccbfa7-1a63-4e96-a74a-5888ceb705b3" containerName="payload-processing" Nov 03 14:43:35.190862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190779 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e51ad9-7c5b-4e9f-88f5-d29e2353bacd" containerName="payload-processing" Nov 03 14:43:35.190862 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.190785 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a646870-36d1-4d20-b25b-59667aecd959" containerName="payload-processing" Nov 03 14:43:35.195327 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.195305 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.218432 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.218407 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb"] Nov 03 14:43:35.330773 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.330733 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.330974 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.330864 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.330974 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.330922 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.331068 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.331020 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.331119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.331096 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.331158 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.331140 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.331214 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.331157 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.331214 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.331187 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.331296 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.331226 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5fkb\" (UniqueName: \"kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-kube-api-access-j5fkb\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.432332 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.432294 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.432544 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.432349 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.432544 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.432378 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.432544 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.432412 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.432544 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.432438 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.432544 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.432473 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.432544 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.432502 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.432544 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.432533 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.432916 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.432558 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5fkb\" (UniqueName: \"kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-kube-api-access-j5fkb\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.432916 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.432805 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.432986 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.432920 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.433225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.433202 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.433376 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.433251 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.433376 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.433333 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.435740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.435720 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.435866 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.435851 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.454041 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.453974 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.454174 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.454093 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5fkb\" (UniqueName: \"kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-kube-api-access-j5fkb\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.508051 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.508019 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:35.846243 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.846219 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb"] Nov 03 14:43:35.848238 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:43:35.848205 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d645ad1_0db6_4b31_9bca_791f72c92cf3.slice/crio-99488ce745ab26b065f416a589b82f192ec241abb5c861a57963d67217e5e172 WatchSource:0}: Error finding container 99488ce745ab26b065f416a589b82f192ec241abb5c861a57963d67217e5e172: Status 404 returned error can't find the container with id 99488ce745ab26b065f416a589b82f192ec241abb5c861a57963d67217e5e172 Nov 03 14:43:35.850543 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.850510 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:43:35.850687 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.850598 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:43:35.850767 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:35.850675 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:43:36.103104 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:36.103020 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" event={"ID":"1d645ad1-0db6-4b31-9bca-791f72c92cf3","Type":"ContainerStarted","Data":"66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559"} Nov 03 14:43:36.103277 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:36.103133 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" event={"ID":"1d645ad1-0db6-4b31-9bca-791f72c92cf3","Type":"ContainerStarted","Data":"99488ce745ab26b065f416a589b82f192ec241abb5c861a57963d67217e5e172"} Nov 03 14:43:36.150906 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:36.150848 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" podStartSLOduration=1.150828305 podStartE2EDuration="1.150828305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:43:36.148789239 +0000 UTC m=+1891.707425725" watchObservedRunningTime="2026-11-03 14:43:36.150828305 +0000 UTC m=+1891.709464790" Nov 03 14:43:36.508469 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:36.508431 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:36.510211 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:36.510186 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 14:43:36.510347 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:36.510233 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" podUID="1d645ad1-0db6-4b31-9bca-791f72c92cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 14:43:37.508948 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:37.508914 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 14:43:37.509344 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:37.508978 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" podUID="1d645ad1-0db6-4b31-9bca-791f72c92cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 14:43:38.509321 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:38.509280 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 14:43:38.509888 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:38.509345 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" podUID="1d645ad1-0db6-4b31-9bca-791f72c92cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 14:43:39.509329 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:39.509292 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 14:43:39.509831 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:39.509355 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" podUID="1d645ad1-0db6-4b31-9bca-791f72c92cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 14:43:40.508903 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:40.508870 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 14:43:40.509063 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:40.508921 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" podUID="1d645ad1-0db6-4b31-9bca-791f72c92cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 14:43:41.509186 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:41.509145 6389 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 14:43:41.509599 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:41.509200 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" podUID="1d645ad1-0db6-4b31-9bca-791f72c92cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 14:43:42.512630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:42.512579 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:42.513099 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:42.513081 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:42.513809 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:43:42.513793 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:43:58.373197 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:43:58.373100 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:44:16.281800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:16.281757 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8"] Nov 03 14:44:16.282255 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:16.281976 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" podUID="625c5998-e17c-4a6e-b885-84d9bd5b41a6" containerName="sim" containerID="cri-o://87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7" gracePeriod=30 Nov 03 14:44:16.630926 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:16.630902 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" Nov 03 14:44:16.792258 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:16.792214 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625c5998-e17c-4a6e-b885-84d9bd5b41a6-kube-api-access-m4966\" (UniqueName: \"kubernetes.io/projected/625c5998-e17c-4a6e-b885-84d9bd5b41a6-kube-api-access-m4966\") pod \"625c5998-e17c-4a6e-b885-84d9bd5b41a6\" (UID: \"625c5998-e17c-4a6e-b885-84d9bd5b41a6\") " Nov 03 14:44:16.792459 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:16.792325 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625c5998-e17c-4a6e-b885-84d9bd5b41a6-tls\" (UniqueName: \"kubernetes.io/secret/625c5998-e17c-4a6e-b885-84d9bd5b41a6-tls\") pod \"625c5998-e17c-4a6e-b885-84d9bd5b41a6\" (UID: \"625c5998-e17c-4a6e-b885-84d9bd5b41a6\") " Nov 03 14:44:16.794716 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:16.794691 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625c5998-e17c-4a6e-b885-84d9bd5b41a6-tls" pod "625c5998-e17c-4a6e-b885-84d9bd5b41a6" (UID: "625c5998-e17c-4a6e-b885-84d9bd5b41a6"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:44:16.794886 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:16.794865 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625c5998-e17c-4a6e-b885-84d9bd5b41a6-kube-api-access-m4966" pod "625c5998-e17c-4a6e-b885-84d9bd5b41a6" (UID: "625c5998-e17c-4a6e-b885-84d9bd5b41a6"). InnerVolumeSpecName "kube-api-access-m4966". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:44:16.894697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:16.894281 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4966\" (UniqueName: \"kubernetes.io/projected/625c5998-e17c-4a6e-b885-84d9bd5b41a6-kube-api-access-m4966\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:44:16.894697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:16.894321 6389 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/625c5998-e17c-4a6e-b885-84d9bd5b41a6-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:44:17.355035 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:17.355003 6389 generic.go:356] "Generic (PLEG): container finished" podID="625c5998-e17c-4a6e-b885-84d9bd5b41a6" containerID="87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7" exitCode=0 Nov 03 14:44:17.355485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:17.355048 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" Nov 03 14:44:17.355485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:17.355082 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" event={"ID":"625c5998-e17c-4a6e-b885-84d9bd5b41a6","Type":"ContainerDied","Data":"87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7"} Nov 03 14:44:17.355485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:17.355127 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8" event={"ID":"625c5998-e17c-4a6e-b885-84d9bd5b41a6","Type":"ContainerDied","Data":"eeb23d3fc0bf82d706b962149ed0470385031ee2901ec2d57015954cd5ed365b"} Nov 03 14:44:17.355485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:17.355149 6389 scope.go:122] "RemoveContainer" containerID="87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7" Nov 03 14:44:17.365169 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:17.365150 6389 scope.go:122] "RemoveContainer" containerID="87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7" Nov 03 14:44:17.365456 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:44:17.365432 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7\": container with ID starting with 87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7 not found: ID does not exist" containerID="87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7" Nov 03 14:44:17.365500 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:17.365467 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7"} err="failed to get container status \"87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7\": rpc error: code = NotFound desc = could not find container \"87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7\": container with ID starting with 87b1a008e403ee418dce6a4497f81670b9e978cc4357de9eb061786fd12bf1e7 not found: ID does not exist" Nov 03 14:44:17.409877 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:17.409848 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8"] Nov 03 14:44:17.422576 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:17.422545 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-96714c/e2e-usage-sim-6293f4-5c7dc84f9d-xvnr8"] Nov 03 14:44:18.617663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:18.617631 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625c5998-e17c-4a6e-b885-84d9bd5b41a6" path="/var/lib/kubelet/pods/625c5998-e17c-4a6e-b885-84d9bd5b41a6/volumes" Nov 03 14:44:28.375219 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:44:28.375181 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:44:33.043516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:33.043473 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd"] Nov 03 14:44:38.044277 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.044233 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" podUID="b24c7af1-4826-4321-a38a-36a980d31b68" containerName="payload-processing" containerID="cri-o://465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce" gracePeriod=25 Nov 03 14:44:38.299403 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.299347 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:44:38.432094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.432034 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-kube-api-access-6vrp9\" (UniqueName: \"kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-kube-api-access-6vrp9\") pod \"b24c7af1-4826-4321-a38a-36a980d31b68\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " Nov 03 14:44:38.432094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.432071 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-routing-overlay-volume\") pod \"b24c7af1-4826-4321-a38a-36a980d31b68\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " Nov 03 14:44:38.432365 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.432128 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-provider-credentials\") pod \"b24c7af1-4826-4321-a38a-36a980d31b68\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " Nov 03 14:44:38.432365 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.432152 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-plugins-config-volume\") pod \"b24c7af1-4826-4321-a38a-36a980d31b68\" (UID: \"b24c7af1-4826-4321-a38a-36a980d31b68\") " Nov 03 14:44:38.432527 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.432498 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-routing-overlay-volume" pod "b24c7af1-4826-4321-a38a-36a980d31b68" (UID: "b24c7af1-4826-4321-a38a-36a980d31b68"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:44:38.432684 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.432627 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-plugins-config-volume" pod "b24c7af1-4826-4321-a38a-36a980d31b68" (UID: "b24c7af1-4826-4321-a38a-36a980d31b68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:44:38.434459 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.434435 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-kube-api-access-6vrp9" pod "b24c7af1-4826-4321-a38a-36a980d31b68" (UID: "b24c7af1-4826-4321-a38a-36a980d31b68"). InnerVolumeSpecName "kube-api-access-6vrp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:44:38.434567 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.434467 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-provider-credentials" pod "b24c7af1-4826-4321-a38a-36a980d31b68" (UID: "b24c7af1-4826-4321-a38a-36a980d31b68"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:44:38.466419 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.466382 6389 generic.go:356] "Generic (PLEG): container finished" podID="b24c7af1-4826-4321-a38a-36a980d31b68" containerID="465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce" exitCode=0 Nov 03 14:44:38.466628 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.466439 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" Nov 03 14:44:38.466628 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.466462 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" event={"ID":"b24c7af1-4826-4321-a38a-36a980d31b68","Type":"ContainerDied","Data":"465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce"} Nov 03 14:44:38.466628 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.466545 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd" event={"ID":"b24c7af1-4826-4321-a38a-36a980d31b68","Type":"ContainerDied","Data":"a777f460d02c5ebcc6f60e9a89caba14fbb50eb09abe7fd5360236e357882e32"} Nov 03 14:44:38.466628 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.466568 6389 scope.go:122] "RemoveContainer" containerID="465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce" Nov 03 14:44:38.476799 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.476780 6389 scope.go:122] "RemoveContainer" containerID="465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce" Nov 03 14:44:38.477083 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:44:38.477060 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce\": container with ID starting with 465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce not found: ID does not exist" containerID="465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce" Nov 03 14:44:38.477133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.477096 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce"} err="failed to get container status \"465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce\": rpc error: code = NotFound desc = could not find container \"465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce\": container with ID starting with 465ecb54928dc82c649bf118a4819da3b503a79d0cf20d3ed3f1fd74b890fcce not found: ID does not exist" Nov 03 14:44:38.522448 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.522420 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd"] Nov 03 14:44:38.528415 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.528392 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5dff86hsd"] Nov 03 14:44:38.533203 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.533183 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vrp9\" (UniqueName: \"kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-kube-api-access-6vrp9\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:44:38.533263 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.533206 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:44:38.533263 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.533217 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b24c7af1-4826-4321-a38a-36a980d31b68-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:44:38.533263 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.533226 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24c7af1-4826-4321-a38a-36a980d31b68-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:44:38.617034 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:44:38.616965 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24c7af1-4826-4321-a38a-36a980d31b68" path="/var/lib/kubelet/pods/b24c7af1-4826-4321-a38a-36a980d31b68/volumes" Nov 03 14:44:58.374138 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:44:58.374094 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:45:23.025966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.025932 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t"] Nov 03 14:45:23.026683 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.026185 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" podUID="9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9" containerName="maas-api" containerID="cri-o://6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b" gracePeriod=30 Nov 03 14:45:23.336860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.336829 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:45:23.402827 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.402793 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-metrics-certs\") pod \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " Nov 03 14:45:23.403005 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.402836 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-maas-api-tls\") pod \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " Nov 03 14:45:23.403005 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.402882 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-openshift-service-ca\") pod \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " Nov 03 14:45:23.403005 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.402911 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-kube-api-access-h8phq\" (UniqueName: \"kubernetes.io/projected/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-kube-api-access-h8phq\") pod \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\" (UID: \"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9\") " Nov 03 14:45:23.403307 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.403281 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-openshift-service-ca" pod "9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9" (UID: "9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:23.405258 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.405227 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-metrics-certs" pod "9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9" (UID: "9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:45:23.405380 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.405346 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-kube-api-access-h8phq" pod "9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9" (UID: "9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9"). InnerVolumeSpecName "kube-api-access-h8phq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:23.405380 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.405281 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-maas-api-tls" pod "9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9" (UID: "9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:45:23.510551 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.505254 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:23.510551 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.505291 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:23.510551 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.505306 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:23.510551 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.505322 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8phq\" (UniqueName: \"kubernetes.io/projected/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9-kube-api-access-h8phq\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:23.724557 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.724521 6389 generic.go:356] "Generic (PLEG): container finished" podID="9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9" containerID="6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b" exitCode=0 Nov 03 14:45:23.724825 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.724569 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" event={"ID":"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9","Type":"ContainerDied","Data":"6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b"} Nov 03 14:45:23.724825 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.724633 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" event={"ID":"9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9","Type":"ContainerDied","Data":"ca5cce4b27c928225bae2630f3d7225ccbc6387ae6494e1db90cfefac3a15034"} Nov 03 14:45:23.724825 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.724656 6389 scope.go:122] "RemoveContainer" containerID="6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b" Nov 03 14:45:23.724825 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.724571 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t" Nov 03 14:45:23.735757 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.735737 6389 scope.go:122] "RemoveContainer" containerID="6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b" Nov 03 14:45:23.736022 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:45:23.736003 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b\": container with ID starting with 6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b not found: ID does not exist" containerID="6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b" Nov 03 14:45:23.736089 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.736034 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b"} err="failed to get container status \"6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b\": rpc error: code = NotFound desc = could not find container \"6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b\": container with ID starting with 6b79362fd6dac028982b4cb819658d8353d17d7f5b128c651b8a6542e0777b3b not found: ID does not exist" Nov 03 14:45:23.781946 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.781908 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t"] Nov 03 14:45:23.787231 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:23.787206 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-331c9e-5cc5f567fb-n667t"] Nov 03 14:45:24.617911 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:24.617880 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9" path="/var/lib/kubelet/pods/9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9/volumes" Nov 03 14:45:28.373282 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:45:28.373243 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:45:31.693503 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:31.693461 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr"] Nov 03 14:45:31.694057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:31.693729 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" containerID="cri-o://b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0" gracePeriod=30 Nov 03 14:45:36.954065 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:36.954038 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:45:37.042841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.042763 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-data\") pod \"a1e41d26-58b4-42ce-a010-e2fa486614d1\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " Nov 03 14:45:37.042841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.042807 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-kube-api-access-8fccn\" (UniqueName: \"kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-kube-api-access-8fccn\") pod \"a1e41d26-58b4-42ce-a010-e2fa486614d1\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " Nov 03 14:45:37.042841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.042830 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e41d26-58b4-42ce-a010-e2fa486614d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e41d26-58b4-42ce-a010-e2fa486614d1-istiod-ca-cert\") pod \"a1e41d26-58b4-42ce-a010-e2fa486614d1\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " Nov 03 14:45:37.043156 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.042911 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-socket\") pod \"a1e41d26-58b4-42ce-a010-e2fa486614d1\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " Nov 03 14:45:37.043156 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.042928 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-token\" (UniqueName: \"kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-token\") pod \"a1e41d26-58b4-42ce-a010-e2fa486614d1\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " Nov 03 14:45:37.043156 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.042985 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-credential-socket\") pod \"a1e41d26-58b4-42ce-a010-e2fa486614d1\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " Nov 03 14:45:37.043156 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043040 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-envoy\") pod \"a1e41d26-58b4-42ce-a010-e2fa486614d1\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " Nov 03 14:45:37.043156 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043059 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-podinfo\") pod \"a1e41d26-58b4-42ce-a010-e2fa486614d1\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " Nov 03 14:45:37.043156 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043075 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-data" pod "a1e41d26-58b4-42ce-a010-e2fa486614d1" (UID: "a1e41d26-58b4-42ce-a010-e2fa486614d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:37.043156 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043089 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-certs\") pod \"a1e41d26-58b4-42ce-a010-e2fa486614d1\" (UID: \"a1e41d26-58b4-42ce-a010-e2fa486614d1\") " Nov 03 14:45:37.043522 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043305 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-socket" pod "a1e41d26-58b4-42ce-a010-e2fa486614d1" (UID: "a1e41d26-58b4-42ce-a010-e2fa486614d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:37.043522 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043330 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-certs" pod "a1e41d26-58b4-42ce-a010-e2fa486614d1" (UID: "a1e41d26-58b4-42ce-a010-e2fa486614d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:37.043522 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043404 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e41d26-58b4-42ce-a010-e2fa486614d1-istiod-ca-cert" pod "a1e41d26-58b4-42ce-a010-e2fa486614d1" (UID: "a1e41d26-58b4-42ce-a010-e2fa486614d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:37.043522 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043471 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-credential-socket" pod "a1e41d26-58b4-42ce-a010-e2fa486614d1" (UID: "a1e41d26-58b4-42ce-a010-e2fa486614d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:37.043919 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043587 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.043919 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043600 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.043919 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043639 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.043919 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043651 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.043919 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.043658 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e41d26-58b4-42ce-a010-e2fa486614d1-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.045339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.045312 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-kube-api-access-8fccn" pod "a1e41d26-58b4-42ce-a010-e2fa486614d1" (UID: "a1e41d26-58b4-42ce-a010-e2fa486614d1"). InnerVolumeSpecName "kube-api-access-8fccn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:37.045339 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.045324 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-token" pod "a1e41d26-58b4-42ce-a010-e2fa486614d1" (UID: "a1e41d26-58b4-42ce-a010-e2fa486614d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:37.045477 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.045374 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-podinfo" pod "a1e41d26-58b4-42ce-a010-e2fa486614d1" (UID: "a1e41d26-58b4-42ce-a010-e2fa486614d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:45:37.045525 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.045502 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-envoy" pod "a1e41d26-58b4-42ce-a010-e2fa486614d1" (UID: "a1e41d26-58b4-42ce-a010-e2fa486614d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:37.144691 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.144654 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.144691 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.144682 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.144691 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.144691 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fccn\" (UniqueName: \"kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-kube-api-access-8fccn\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.144925 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.144701 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e41d26-58b4-42ce-a010-e2fa486614d1-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.799813 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.799779 6389 generic.go:356] "Generic (PLEG): container finished" podID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerID="b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0" exitCode=0 Nov 03 14:45:37.800018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.799827 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" Nov 03 14:45:37.800018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.799868 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" event={"ID":"a1e41d26-58b4-42ce-a010-e2fa486614d1","Type":"ContainerDied","Data":"b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0"} Nov 03 14:45:37.800018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.799925 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr" event={"ID":"a1e41d26-58b4-42ce-a010-e2fa486614d1","Type":"ContainerDied","Data":"3a5ecdeb00849d0d6f9aadf04a6fdd2ee37ed6cea5297d3a4c72a4eeab17da08"} Nov 03 14:45:37.800018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.799954 6389 scope.go:122] "RemoveContainer" containerID="b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0" Nov 03 14:45:37.809840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.809818 6389 scope.go:122] "RemoveContainer" containerID="88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563" Nov 03 14:45:37.818400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.818203 6389 scope.go:122] "RemoveContainer" containerID="b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0" Nov 03 14:45:37.818504 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:45:37.818486 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0\": container with ID starting with b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0 not found: ID does not exist" containerID="b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0" Nov 03 14:45:37.818550 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.818513 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0"} err="failed to get container status \"b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0\": rpc error: code = NotFound desc = could not find container \"b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0\": container with ID starting with b609cdda69b1fdd8ab151ffa301da5db299d3f9df592065f5cdc9b830ef0bfb0 not found: ID does not exist" Nov 03 14:45:37.818550 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.818531 6389 scope.go:122] "RemoveContainer" containerID="88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563" Nov 03 14:45:37.818808 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:45:37.818789 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563\": container with ID starting with 88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563 not found: ID does not exist" containerID="88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563" Nov 03 14:45:37.818857 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.818814 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563"} err="failed to get container status \"88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563\": rpc error: code = NotFound desc = could not find container \"88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563\": container with ID starting with 88ebde2be1a59bc507a4d159d296a9c0899af16a6c8a23f12ffd9c877728a563 not found: ID does not exist" Nov 03 14:45:37.872068 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.872036 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr"] Nov 03 14:45:37.877535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:37.877512 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-9cnhr"] Nov 03 14:45:38.618302 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:38.618267 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" path="/var/lib/kubelet/pods/a1e41d26-58b4-42ce-a010-e2fa486614d1/volumes" Nov 03 14:45:41.506859 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.506826 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j"] Nov 03 14:45:41.507342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.507084 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" podUID="b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70" containerName="maas-api" containerID="cri-o://a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41" gracePeriod=30 Nov 03 14:45:41.800657 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.798989 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:45:41.821016 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.820980 6389 generic.go:356] "Generic (PLEG): container finished" podID="b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70" containerID="a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41" exitCode=0 Nov 03 14:45:41.821207 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.821033 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" Nov 03 14:45:41.821207 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.821076 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" event={"ID":"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70","Type":"ContainerDied","Data":"a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41"} Nov 03 14:45:41.821207 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.821137 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j" event={"ID":"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70","Type":"ContainerDied","Data":"89db4e756cfcc9f9fa28f3d9571ae2d885cdc25c8b8d02d56b0ca096559268fa"} Nov 03 14:45:41.821207 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.821169 6389 scope.go:122] "RemoveContainer" containerID="a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41" Nov 03 14:45:41.830731 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.830709 6389 scope.go:122] "RemoveContainer" containerID="a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41" Nov 03 14:45:41.831053 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:45:41.831030 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41\": container with ID starting with a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41 not found: ID does not exist" containerID="a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41" Nov 03 14:45:41.831116 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.831063 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41"} err="failed to get container status \"a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41\": rpc error: code = NotFound desc = could not find container \"a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41\": container with ID starting with a6df33d947d9e095058cdd668f38a3ea22136ed055e876e1546df1d06daa3b41 not found: ID does not exist" Nov 03 14:45:41.903835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.903802 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-kube-api-access-x2c6j\" (UniqueName: \"kubernetes.io/projected/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-kube-api-access-x2c6j\") pod \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " Nov 03 14:45:41.904015 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.903853 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-metrics-certs\") pod \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " Nov 03 14:45:41.904015 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.903916 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-openshift-service-ca\") pod \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " Nov 03 14:45:41.904015 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.903973 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-maas-api-tls\") pod \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\" (UID: \"b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70\") " Nov 03 14:45:41.904374 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.904343 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-openshift-service-ca" pod "b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70" (UID: "b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:41.906059 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.906033 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-kube-api-access-x2c6j" pod "b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70" (UID: "b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70"). InnerVolumeSpecName "kube-api-access-x2c6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:41.906178 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.906065 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-maas-api-tls" pod "b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70" (UID: "b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:45:41.906178 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:41.906121 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-metrics-certs" pod "b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70" (UID: "b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:45:42.005821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:42.005787 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2c6j\" (UniqueName: \"kubernetes.io/projected/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-kube-api-access-x2c6j\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:42.005821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:42.005822 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:42.006035 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:42.005839 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:42.006035 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:42.005855 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:42.168548 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:42.168511 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j"] Nov 03 14:45:42.173768 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:42.173738 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4361f-69f67d9ccf-9mp4j"] Nov 03 14:45:42.617415 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:42.617384 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70" path="/var/lib/kubelet/pods/b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70/volumes" Nov 03 14:45:48.915888 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:48.915841 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb"] Nov 03 14:45:48.916739 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:48.916714 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" podUID="1d645ad1-0db6-4b31-9bca-791f72c92cf3" containerName="istio-proxy" containerID="cri-o://66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559" gracePeriod=30 Nov 03 14:45:48.928838 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:48.928807 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp"] Nov 03 14:45:48.929084 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:48.929053 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" containerName="istio-proxy" containerID="cri-o://a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400" gracePeriod=30 Nov 03 14:45:50.690496 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:50.690471 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-k9lp2_bf69a67f-797a-4489-a929-789d47efc69c/kuadrant-console-plugin/0.log" Nov 03 14:45:52.693966 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:52.693932 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5f464594c6-562pp_428b2a8c-5dc4-4b29-a535-35b8b20cf721/manager/0.log" Nov 03 14:45:53.071515 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:53.071429 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qdflj_f884e514-a7a4-459f-aba9-62f7ce76671a/manager/0.log" Nov 03 14:45:53.295179 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:53.295137 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw"] Nov 03 14:45:53.295372 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:53.295351 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" containerID="cri-o://439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac" gracePeriod=30 Nov 03 14:45:53.697224 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:53.697174 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dfdc587c5-jcn8k_93952091-e0db-4779-9d1a-fab69c52f73f/manager/0.log" Nov 03 14:45:54.181098 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.181074 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:45:54.207401 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.207343 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:45:54.222579 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.222556 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-socket\") pod \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " Nov 03 14:45:54.222666 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.222587 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-token\" (UniqueName: \"kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-token\") pod \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " Nov 03 14:45:54.222666 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.222639 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istiod-ca-cert\") pod \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " Nov 03 14:45:54.222772 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.222673 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-certs\") pod \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " Nov 03 14:45:54.222772 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.222729 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-podinfo\") pod \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " Nov 03 14:45:54.222772 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.222761 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-kube-api-access-j5fkb\" (UniqueName: \"kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-kube-api-access-j5fkb\") pod \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " Nov 03 14:45:54.222922 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.222794 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-credential-socket\") pod \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " Nov 03 14:45:54.222922 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.222835 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-socket" pod "1d645ad1-0db6-4b31-9bca-791f72c92cf3" (UID: "1d645ad1-0db6-4b31-9bca-791f72c92cf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.222922 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.222918 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-envoy\") pod \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " Nov 03 14:45:54.223089 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.222955 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-data\") pod \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\" (UID: \"1d645ad1-0db6-4b31-9bca-791f72c92cf3\") " Nov 03 14:45:54.223089 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.222974 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-certs" pod "1d645ad1-0db6-4b31-9bca-791f72c92cf3" (UID: "1d645ad1-0db6-4b31-9bca-791f72c92cf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.223216 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.223053 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istiod-ca-cert" pod "1d645ad1-0db6-4b31-9bca-791f72c92cf3" (UID: "1d645ad1-0db6-4b31-9bca-791f72c92cf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:54.223216 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.223130 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-credential-socket" pod "1d645ad1-0db6-4b31-9bca-791f72c92cf3" (UID: "1d645ad1-0db6-4b31-9bca-791f72c92cf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.223359 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.223339 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-data" pod "1d645ad1-0db6-4b31-9bca-791f72c92cf3" (UID: "1d645ad1-0db6-4b31-9bca-791f72c92cf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.223601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.223531 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.223601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.223557 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.223601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.223571 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.223601 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.223584 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.224973 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.224951 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-token" pod "1d645ad1-0db6-4b31-9bca-791f72c92cf3" (UID: "1d645ad1-0db6-4b31-9bca-791f72c92cf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:54.225094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.225076 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-envoy" pod "1d645ad1-0db6-4b31-9bca-791f72c92cf3" (UID: "1d645ad1-0db6-4b31-9bca-791f72c92cf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.225094 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.225082 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-podinfo" pod "1d645ad1-0db6-4b31-9bca-791f72c92cf3" (UID: "1d645ad1-0db6-4b31-9bca-791f72c92cf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:45:54.225169 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.225101 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-kube-api-access-j5fkb" pod "1d645ad1-0db6-4b31-9bca-791f72c92cf3" (UID: "1d645ad1-0db6-4b31-9bca-791f72c92cf3"). InnerVolumeSpecName "kube-api-access-j5fkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:54.324202 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324170 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6871153-f93d-4cc0-b09b-f8f303706858-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6871153-f93d-4cc0-b09b-f8f303706858-istio-podinfo\") pod \"e6871153-f93d-4cc0-b09b-f8f303706858\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " Nov 03 14:45:54.324391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324239 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-socket\") pod \"e6871153-f93d-4cc0-b09b-f8f303706858\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " Nov 03 14:45:54.324391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324288 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-envoy\") pod \"e6871153-f93d-4cc0-b09b-f8f303706858\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " Nov 03 14:45:54.324391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324335 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-kube-api-access-87f9k\" (UniqueName: \"kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-kube-api-access-87f9k\") pod \"e6871153-f93d-4cc0-b09b-f8f303706858\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " Nov 03 14:45:54.324566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324407 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-certs\") pod \"e6871153-f93d-4cc0-b09b-f8f303706858\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " Nov 03 14:45:54.324566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324476 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6871153-f93d-4cc0-b09b-f8f303706858-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6871153-f93d-4cc0-b09b-f8f303706858-istiod-ca-cert\") pod \"e6871153-f93d-4cc0-b09b-f8f303706858\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " Nov 03 14:45:54.324566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324519 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-istio-token\" (UniqueName: \"kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-istio-token\") pod \"e6871153-f93d-4cc0-b09b-f8f303706858\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " Nov 03 14:45:54.324566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324532 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-socket" pod "e6871153-f93d-4cc0-b09b-f8f303706858" (UID: "e6871153-f93d-4cc0-b09b-f8f303706858"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.324566 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324551 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-data\") pod \"e6871153-f93d-4cc0-b09b-f8f303706858\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " Nov 03 14:45:54.324860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324580 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-credential-socket\") pod \"e6871153-f93d-4cc0-b09b-f8f303706858\" (UID: \"e6871153-f93d-4cc0-b09b-f8f303706858\") " Nov 03 14:45:54.324860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324798 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-certs" pod "e6871153-f93d-4cc0-b09b-f8f303706858" (UID: "e6871153-f93d-4cc0-b09b-f8f303706858"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.324860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324817 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-data" pod "e6871153-f93d-4cc0-b09b-f8f303706858" (UID: "e6871153-f93d-4cc0-b09b-f8f303706858"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.325016 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.324916 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6871153-f93d-4cc0-b09b-f8f303706858-istiod-ca-cert" pod "e6871153-f93d-4cc0-b09b-f8f303706858" (UID: "e6871153-f93d-4cc0-b09b-f8f303706858"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:54.325069 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.325010 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-credential-socket" pod "e6871153-f93d-4cc0-b09b-f8f303706858" (UID: "e6871153-f93d-4cc0-b09b-f8f303706858"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.325114 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.325095 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6871153-f93d-4cc0-b09b-f8f303706858-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.325172 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.325119 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.325172 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.325135 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.325242 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.325166 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.325307 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.325249 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.325344 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.325332 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.325383 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.325345 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d645ad1-0db6-4b31-9bca-791f72c92cf3-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.325383 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.325357 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5fkb\" (UniqueName: \"kubernetes.io/projected/1d645ad1-0db6-4b31-9bca-791f72c92cf3-kube-api-access-j5fkb\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.325383 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.325372 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.326830 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.326809 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6871153-f93d-4cc0-b09b-f8f303706858-istio-podinfo" pod "e6871153-f93d-4cc0-b09b-f8f303706858" (UID: "e6871153-f93d-4cc0-b09b-f8f303706858"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:45:54.326830 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.326809 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-istio-token" pod "e6871153-f93d-4cc0-b09b-f8f303706858" (UID: "e6871153-f93d-4cc0-b09b-f8f303706858"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:54.327133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.327113 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-envoy" pod "e6871153-f93d-4cc0-b09b-f8f303706858" (UID: "e6871153-f93d-4cc0-b09b-f8f303706858"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.327133 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.327115 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-kube-api-access-87f9k" pod "e6871153-f93d-4cc0-b09b-f8f303706858" (UID: "e6871153-f93d-4cc0-b09b-f8f303706858"). InnerVolumeSpecName "kube-api-access-87f9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:54.426663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.426597 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.426663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.426653 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.426663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.426664 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6871153-f93d-4cc0-b09b-f8f303706858-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.426663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.426672 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6871153-f93d-4cc0-b09b-f8f303706858-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.426934 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.426683 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87f9k\" (UniqueName: \"kubernetes.io/projected/e6871153-f93d-4cc0-b09b-f8f303706858-kube-api-access-87f9k\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.886543 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.886450 6389 generic.go:356] "Generic (PLEG): container finished" podID="e6871153-f93d-4cc0-b09b-f8f303706858" containerID="a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400" exitCode=0 Nov 03 14:45:54.886543 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.886507 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" Nov 03 14:45:54.887113 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.886549 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" event={"ID":"e6871153-f93d-4cc0-b09b-f8f303706858","Type":"ContainerDied","Data":"a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400"} Nov 03 14:45:54.887113 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.886638 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp" event={"ID":"e6871153-f93d-4cc0-b09b-f8f303706858","Type":"ContainerDied","Data":"f5365f9dba5ee67515d68326b7220d6ecedd0f3cd9c582ac331ac5e526c3b1a6"} Nov 03 14:45:54.887113 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.886673 6389 scope.go:122] "RemoveContainer" containerID="a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400" Nov 03 14:45:54.888090 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.887965 6389 generic.go:356] "Generic (PLEG): container finished" podID="1d645ad1-0db6-4b31-9bca-791f72c92cf3" containerID="66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559" exitCode=0 Nov 03 14:45:54.888090 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.888019 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" event={"ID":"1d645ad1-0db6-4b31-9bca-791f72c92cf3","Type":"ContainerDied","Data":"66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559"} Nov 03 14:45:54.888090 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.888035 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" Nov 03 14:45:54.888090 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.888051 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb" event={"ID":"1d645ad1-0db6-4b31-9bca-791f72c92cf3","Type":"ContainerDied","Data":"99488ce745ab26b065f416a589b82f192ec241abb5c861a57963d67217e5e172"} Nov 03 14:45:54.895776 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.895761 6389 scope.go:122] "RemoveContainer" containerID="a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400" Nov 03 14:45:54.896046 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:45:54.896029 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400\": container with ID starting with a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400 not found: ID does not exist" containerID="a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400" Nov 03 14:45:54.896125 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.896058 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400"} err="failed to get container status \"a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400\": rpc error: code = NotFound desc = could not find container \"a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400\": container with ID starting with a8389c1b508f656b64ed9adc7ed606dd9ce338bfa0aa5f0eef3917d10d3c3400 not found: ID does not exist" Nov 03 14:45:54.896125 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.896081 6389 scope.go:122] "RemoveContainer" containerID="66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559" Nov 03 14:45:54.904185 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.904163 6389 scope.go:122] "RemoveContainer" containerID="66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559" Nov 03 14:45:54.904431 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:45:54.904415 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559\": container with ID starting with 66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559 not found: ID does not exist" containerID="66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559" Nov 03 14:45:54.904480 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.904438 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559"} err="failed to get container status \"66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559\": rpc error: code = NotFound desc = could not find container \"66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559\": container with ID starting with 66bf682a8d7c0536841469499232944efb9897a8c757bd955a2ae02107382559 not found: ID does not exist" Nov 03 14:45:54.956908 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.956881 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb"] Nov 03 14:45:54.969360 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:54.969334 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-wk9nb"] Nov 03 14:45:55.004294 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:55.004262 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp"] Nov 03 14:45:55.018189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:55.018158 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-dpzhp"] Nov 03 14:45:56.616322 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:56.616293 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d645ad1-0db6-4b31-9bca-791f72c92cf3" path="/var/lib/kubelet/pods/1d645ad1-0db6-4b31-9bca-791f72c92cf3/volumes" Nov 03 14:45:56.616761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:56.616747 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" path="/var/lib/kubelet/pods/e6871153-f93d-4cc0-b09b-f8f303706858/volumes" Nov 03 14:45:58.373567 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:45:58.373526 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:45:58.544384 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.544358 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:45:58.665557 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.665464 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-podinfo\") pod \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " Nov 03 14:45:58.665557 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.665523 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-certs\") pod \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " Nov 03 14:45:58.665816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.665589 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-envoy\") pod \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " Nov 03 14:45:58.665816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.665632 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istiod-ca-cert\") pod \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " Nov 03 14:45:58.665816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.665664 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-data\") pod \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " Nov 03 14:45:58.665816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.665684 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-token\" (UniqueName: \"kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-token\") pod \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " Nov 03 14:45:58.665816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.665721 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-kube-api-access-djnzg\" (UniqueName: \"kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-kube-api-access-djnzg\") pod \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " Nov 03 14:45:58.665816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.665751 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-socket\") pod \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " Nov 03 14:45:58.665816 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.665815 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-credential-socket\") pod \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\" (UID: \"4c1fd640-11de-42e6-8ab7-19ced0bcbb05\") " Nov 03 14:45:58.666178 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.665884 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-certs" pod "4c1fd640-11de-42e6-8ab7-19ced0bcbb05" (UID: "4c1fd640-11de-42e6-8ab7-19ced0bcbb05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.666178 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.666049 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-data" pod "4c1fd640-11de-42e6-8ab7-19ced0bcbb05" (UID: "4c1fd640-11de-42e6-8ab7-19ced0bcbb05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.666280 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.666202 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.666280 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.666222 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.666457 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.666436 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-credential-socket" pod "4c1fd640-11de-42e6-8ab7-19ced0bcbb05" (UID: "4c1fd640-11de-42e6-8ab7-19ced0bcbb05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.666508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.666490 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istiod-ca-cert" pod "4c1fd640-11de-42e6-8ab7-19ced0bcbb05" (UID: "4c1fd640-11de-42e6-8ab7-19ced0bcbb05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:58.666838 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.666808 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-socket" pod "4c1fd640-11de-42e6-8ab7-19ced0bcbb05" (UID: "4c1fd640-11de-42e6-8ab7-19ced0bcbb05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.667986 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.667962 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-envoy" pod "4c1fd640-11de-42e6-8ab7-19ced0bcbb05" (UID: "4c1fd640-11de-42e6-8ab7-19ced0bcbb05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.667986 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.667965 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-podinfo" pod "4c1fd640-11de-42e6-8ab7-19ced0bcbb05" (UID: "4c1fd640-11de-42e6-8ab7-19ced0bcbb05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:45:58.668885 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.668851 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-kube-api-access-djnzg" pod "4c1fd640-11de-42e6-8ab7-19ced0bcbb05" (UID: "4c1fd640-11de-42e6-8ab7-19ced0bcbb05"). InnerVolumeSpecName "kube-api-access-djnzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:58.668960 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.668927 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-token" pod "4c1fd640-11de-42e6-8ab7-19ced0bcbb05" (UID: "4c1fd640-11de-42e6-8ab7-19ced0bcbb05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:58.766793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.766761 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djnzg\" (UniqueName: \"kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-kube-api-access-djnzg\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.766793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.766790 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.766793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.766799 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.767035 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.766808 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.767035 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.766817 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.767035 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.766825 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.767035 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.766833 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1fd640-11de-42e6-8ab7-19ced0bcbb05-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.911174 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.911134 6389 generic.go:356] "Generic (PLEG): container finished" podID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerID="439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac" exitCode=0 Nov 03 14:45:58.911343 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.911189 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" Nov 03 14:45:58.911343 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.911203 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" event={"ID":"4c1fd640-11de-42e6-8ab7-19ced0bcbb05","Type":"ContainerDied","Data":"439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac"} Nov 03 14:45:58.911343 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.911259 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw" event={"ID":"4c1fd640-11de-42e6-8ab7-19ced0bcbb05","Type":"ContainerDied","Data":"3a828edb64d2b3074af3ecdb3bd7651ef793387bdbb2a837039ad8a95e10cb49"} Nov 03 14:45:58.911343 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.911282 6389 scope.go:122] "RemoveContainer" containerID="439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac" Nov 03 14:45:58.920509 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.920494 6389 scope.go:122] "RemoveContainer" containerID="439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac" Nov 03 14:45:58.920800 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:45:58.920780 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac\": container with ID starting with 439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac not found: ID does not exist" containerID="439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac" Nov 03 14:45:58.920889 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.920807 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac"} err="failed to get container status \"439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac\": rpc error: code = NotFound desc = could not find container \"439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac\": container with ID starting with 439e810990739d343ea677a7d10cb30e04a15f6ee12391daa80e703854122dac not found: ID does not exist" Nov 03 14:45:58.970357 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.970325 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw"] Nov 03 14:45:58.982763 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:58.982716 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qbxzw"] Nov 03 14:45:59.100359 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100327 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b"] Nov 03 14:45:59.100764 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100739 6389 cpu_manager.go:414] "containerMap: removing container" podUID="1d645ad1-0db6-4b31-9bca-791f72c92cf3" containerName="istio-proxy" Nov 03 14:45:59.100764 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100759 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d645ad1-0db6-4b31-9bca-791f72c92cf3" containerName="istio-proxy" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100783 6389 cpu_manager.go:414] "containerMap: removing container" podUID="625c5998-e17c-4a6e-b885-84d9bd5b41a6" containerName="sim" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100790 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625c5998-e17c-4a6e-b885-84d9bd5b41a6" containerName="sim" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100811 6389 cpu_manager.go:414] "containerMap: removing container" podUID="9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9" containerName="maas-api" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100818 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9" containerName="maas-api" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100833 6389 cpu_manager.go:414] "containerMap: removing container" podUID="b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70" containerName="maas-api" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100840 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70" containerName="maas-api" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100865 6389 cpu_manager.go:414] "containerMap: removing container" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" containerName="istio-proxy" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100871 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" containerName="istio-proxy" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100889 6389 cpu_manager.go:414] "containerMap: removing container" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100896 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100904 6389 cpu_manager.go:414] "containerMap: removing container" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100911 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100921 6389 cpu_manager.go:414] "containerMap: removing container" podUID="b24c7af1-4826-4321-a38a-36a980d31b68" containerName="payload-processing" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100928 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24c7af1-4826-4321-a38a-36a980d31b68" containerName="payload-processing" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100943 6389 cpu_manager.go:414] "containerMap: removing container" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" Nov 03 14:45:59.100954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.100951 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" Nov 03 14:45:59.101740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.101047 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" Nov 03 14:45:59.101740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.101058 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24c7af1-4826-4321-a38a-36a980d31b68" containerName="payload-processing" Nov 03 14:45:59.101740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.101066 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7fe6f4-8d7d-4b1e-b0ec-9963cd72cbc9" containerName="maas-api" Nov 03 14:45:59.101740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.101076 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e41d26-58b4-42ce-a010-e2fa486614d1" containerName="istio-proxy" Nov 03 14:45:59.101740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.101085 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="b87a0036-e3fd-4aa5-9c1d-c4cf4a599c70" containerName="maas-api" Nov 03 14:45:59.101740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.101094 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d645ad1-0db6-4b31-9bca-791f72c92cf3" containerName="istio-proxy" Nov 03 14:45:59.101740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.101107 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="625c5998-e17c-4a6e-b885-84d9bd5b41a6" containerName="sim" Nov 03 14:45:59.101740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.101115 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6871153-f93d-4cc0-b09b-f8f303706858" containerName="istio-proxy" Nov 03 14:45:59.101740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.101125 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" containerName="istio-proxy" Nov 03 14:45:59.105910 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.105890 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.128763 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.128735 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8c02610e-openshift-default-dockercfg-x674k\"" Nov 03 14:45:59.131003 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.130975 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b"] Nov 03 14:45:59.171199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.171115 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwjmg\" (UniqueName: \"kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-kube-api-access-zwjmg\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.171199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.171154 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-credential-socket\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.171199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.171175 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-certs\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.171426 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.171268 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-token\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.171426 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.171309 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istiod-ca-cert\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.171426 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.171335 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-data\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.171579 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.171428 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-socket\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.171579 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.171447 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-envoy\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.171579 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.171549 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-podinfo\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272185 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272144 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwjmg\" (UniqueName: \"kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-kube-api-access-zwjmg\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272185 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272185 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-credential-socket\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272203 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-certs\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272236 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-token\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272263 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istiod-ca-cert\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272288 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-data\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272357 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-socket\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272388 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-envoy\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272438 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272429 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-podinfo\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272775 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272684 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-credential-socket\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272865 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272847 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-certs\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272909 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-socket\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.272964 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.272919 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-data\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.273095 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.273070 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istiod-ca-cert\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.275057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.275037 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-envoy\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.275184 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.275163 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-podinfo\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.294682 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.294659 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-token\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.294824 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.294804 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwjmg\" (UniqueName: \"kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-kube-api-access-zwjmg\") pod \"e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.419344 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.419308 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:45:59.746260 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.746228 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b"] Nov 03 14:45:59.757318 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:45:59.757288 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873eaa3a_cbd6_4b40_881a_49d009f2f4f4.slice/crio-efdeee560b24ebb5bc31d65385594a36d045eb02de3a8f0378e29fe6b7ad98b9 WatchSource:0}: Error finding container efdeee560b24ebb5bc31d65385594a36d045eb02de3a8f0378e29fe6b7ad98b9: Status 404 returned error can't find the container with id efdeee560b24ebb5bc31d65385594a36d045eb02de3a8f0378e29fe6b7ad98b9 Nov 03 14:45:59.759314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.759281 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:45:59.759412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.759344 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:45:59.759412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.759390 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:45:59.918721 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.918681 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" event={"ID":"873eaa3a-cbd6-4b40-881a-49d009f2f4f4","Type":"ContainerStarted","Data":"c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185"} Nov 03 14:45:59.918927 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.918746 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" event={"ID":"873eaa3a-cbd6-4b40-881a-49d009f2f4f4","Type":"ContainerStarted","Data":"efdeee560b24ebb5bc31d65385594a36d045eb02de3a8f0378e29fe6b7ad98b9"} Nov 03 14:45:59.969142 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:45:59.969075 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" podStartSLOduration=0.969055021 podStartE2EDuration="969.055021ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:45:59.967241752 +0000 UTC m=+2035.525878237" watchObservedRunningTime="2026-11-03 14:45:59.969055021 +0000 UTC m=+2035.527691508" Nov 03 14:46:00.419810 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:00.419779 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:46:00.424844 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:00.424817 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:46:00.617304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:00.617264 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c1fd640-11de-42e6-8ab7-19ced0bcbb05" path="/var/lib/kubelet/pods/4c1fd640-11de-42e6-8ab7-19ced0bcbb05/volumes" Nov 03 14:46:00.924832 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:00.924805 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:46:00.925696 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:00.925677 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:46:03.487023 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.486985 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw"] Nov 03 14:46:03.490898 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.490877 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.512721 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.512689 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8c02610e-serving-cert\"" Nov 03 14:46:03.524891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.524855 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw"] Nov 03 14:46:03.611522 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.611493 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-maas-api-tls\") pod \"maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.611708 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.611531 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-openshift-service-ca\") pod \"maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.611708 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.611687 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-metrics-certs\") pod \"maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.611828 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.611708 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8wq\" (UniqueName: \"kubernetes.io/projected/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-kube-api-access-hd8wq\") pod \"maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.713244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.713210 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-maas-api-tls\") pod \"maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.713244 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.713251 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-openshift-service-ca\") pod \"maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.713521 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.713304 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-metrics-certs\") pod \"maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.713521 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.713324 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8wq\" (UniqueName: \"kubernetes.io/projected/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-kube-api-access-hd8wq\") pod \"maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.714093 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.714072 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-openshift-service-ca\") pod \"maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.716046 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.716018 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-maas-api-tls\") pod \"maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.716154 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.716048 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-metrics-certs\") pod \"maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.734981 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.734945 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8wq\" (UniqueName: \"kubernetes.io/projected/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-kube-api-access-hd8wq\") pod \"maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:03.805057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:03.804972 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:04.106950 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:04.106798 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw"] Nov 03 14:46:04.109117 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:46:04.109091 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13cdb3ca_71a4_4a86_8bba_3c3ad895c12c.slice/crio-58968d51b9bf408b6cac220a0da5f6e6c93e0aa9b94cd9dcd3cb4fa7277eed11 WatchSource:0}: Error finding container 58968d51b9bf408b6cac220a0da5f6e6c93e0aa9b94cd9dcd3cb4fa7277eed11: Status 404 returned error can't find the container with id 58968d51b9bf408b6cac220a0da5f6e6c93e0aa9b94cd9dcd3cb4fa7277eed11 Nov 03 14:46:04.110477 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:04.110458 6389 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:46:04.946290 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:04.946250 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" event={"ID":"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c","Type":"ContainerStarted","Data":"135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9"} Nov 03 14:46:04.946723 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:04.946407 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:04.946723 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:04.946443 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" event={"ID":"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c","Type":"ContainerStarted","Data":"58968d51b9bf408b6cac220a0da5f6e6c93e0aa9b94cd9dcd3cb4fa7277eed11"} Nov 03 14:46:04.990355 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:04.990291 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" podStartSLOduration=1.538799509 podStartE2EDuration="1.990277821s" totalImagesPullingTime="451.478312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:46:04.988861683 +0000 UTC m=+2040.547498168" watchObservedRunningTime="2026-11-03 14:46:04.990277821 +0000 UTC m=+2040.548914305" Nov 03 14:46:10.957711 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:10.957680 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:16.924331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:16.923744 6389 patch_prober.go:29] interesting pod/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": read tcp 10.134.0.2:57786->10.134.0.138:15021: read: connection reset by peer" start-of-body= Nov 03 14:46:16.924331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:16.923802 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": read tcp 10.134.0.2:57786->10.134.0.138:15021: read: connection reset by peer" Nov 03 14:46:17.007999 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:17.007971 6389 generic.go:356] "Generic (PLEG): container finished" podID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerID="c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185" exitCode=0 Nov 03 14:46:17.008143 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:17.008046 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" event={"ID":"873eaa3a-cbd6-4b40-881a-49d009f2f4f4","Type":"ContainerDied","Data":"c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185"} Nov 03 14:46:17.008404 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:17.008391 6389 scope.go:122] "RemoveContainer" containerID="c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185" Nov 03 14:46:17.009541 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:17.009514 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:46:17.009647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:17.009585 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:46:17.009647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:17.009642 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:46:18.014699 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:18.014662 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" event={"ID":"873eaa3a-cbd6-4b40-881a-49d009f2f4f4","Type":"ContainerStarted","Data":"538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a"} Nov 03 14:46:18.426642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:18.425780 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:46:18.426642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:18.426292 6389 patch_prober.go:29] interesting pod/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 14:46:18.426642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:18.426386 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 14:46:19.435633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:19.431001 6389 patch_prober.go:29] interesting pod/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 14:46:19.435633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:19.431063 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 14:46:20.444071 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:20.444031 6389 patch_prober.go:29] interesting pod/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 14:46:20.444581 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:20.444096 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 14:46:21.443785 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:21.443737 6389 patch_prober.go:29] interesting pod/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 14:46:21.444004 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:21.443806 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 14:46:22.435760 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:22.435723 6389 patch_prober.go:29] interesting pod/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 14:46:22.436436 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:22.436402 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 14:46:23.423896 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:23.423867 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:46:23.424161 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:23.424140 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:46:23.424834 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:23.424818 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:46:28.235857 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.235829 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw"] Nov 03 14:46:28.236287 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.236036 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" podUID="13cdb3ca-71a4-4a86-8bba-3c3ad895c12c" containerName="maas-api" containerID="cri-o://135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9" gracePeriod=30 Nov 03 14:46:28.373730 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:46:28.373690 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:46:28.490681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.490624 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:28.564097 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.564062 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-metrics-certs\") pod \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " Nov 03 14:46:28.564281 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.564202 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-openshift-service-ca\") pod \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " Nov 03 14:46:28.564281 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.564251 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-kube-api-access-hd8wq\" (UniqueName: \"kubernetes.io/projected/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-kube-api-access-hd8wq\") pod \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " Nov 03 14:46:28.564406 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.564288 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-maas-api-tls\") pod \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\" (UID: \"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c\") " Nov 03 14:46:28.564602 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.564559 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-openshift-service-ca" pod "13cdb3ca-71a4-4a86-8bba-3c3ad895c12c" (UID: "13cdb3ca-71a4-4a86-8bba-3c3ad895c12c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:46:28.566589 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.566560 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-maas-api-tls" pod "13cdb3ca-71a4-4a86-8bba-3c3ad895c12c" (UID: "13cdb3ca-71a4-4a86-8bba-3c3ad895c12c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:46:28.566798 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.566595 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-metrics-certs" pod "13cdb3ca-71a4-4a86-8bba-3c3ad895c12c" (UID: "13cdb3ca-71a4-4a86-8bba-3c3ad895c12c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:46:28.566798 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.566705 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-kube-api-access-hd8wq" pod "13cdb3ca-71a4-4a86-8bba-3c3ad895c12c" (UID: "13cdb3ca-71a4-4a86-8bba-3c3ad895c12c"). InnerVolumeSpecName "kube-api-access-hd8wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:46:28.667957 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.667910 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:28.667957 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.667949 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:28.667957 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.667965 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:28.668226 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:28.667979 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd8wq\" (UniqueName: \"kubernetes.io/projected/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c-kube-api-access-hd8wq\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:29.093092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:29.093058 6389 generic.go:356] "Generic (PLEG): container finished" podID="13cdb3ca-71a4-4a86-8bba-3c3ad895c12c" containerID="135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9" exitCode=0 Nov 03 14:46:29.093285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:29.093114 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" Nov 03 14:46:29.093285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:29.093150 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" event={"ID":"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c","Type":"ContainerDied","Data":"135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9"} Nov 03 14:46:29.093285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:29.093204 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw" event={"ID":"13cdb3ca-71a4-4a86-8bba-3c3ad895c12c","Type":"ContainerDied","Data":"58968d51b9bf408b6cac220a0da5f6e6c93e0aa9b94cd9dcd3cb4fa7277eed11"} Nov 03 14:46:29.093285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:29.093225 6389 scope.go:122] "RemoveContainer" containerID="135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9" Nov 03 14:46:29.101909 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:29.101891 6389 scope.go:122] "RemoveContainer" containerID="135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9" Nov 03 14:46:29.102189 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:46:29.102153 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9\": container with ID starting with 135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9 not found: ID does not exist" containerID="135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9" Nov 03 14:46:29.102240 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:29.102197 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9"} err="failed to get container status \"135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9\": rpc error: code = NotFound desc = could not find container \"135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9\": container with ID starting with 135dc58274aa9315788767f561400d6747b0248d3422024da523bb04481ab5e9 not found: ID does not exist" Nov 03 14:46:29.147508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:29.147476 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw"] Nov 03 14:46:29.159800 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:29.159771 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c02610e-7d7c7f9fd8-2lxtw"] Nov 03 14:46:30.627302 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:30.627254 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13cdb3ca-71a4-4a86-8bba-3c3ad895c12c" path="/var/lib/kubelet/pods/13cdb3ca-71a4-4a86-8bba-3c3ad895c12c/volumes" Nov 03 14:46:33.777073 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:33.777041 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b"] Nov 03 14:46:33.777517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:33.777267 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" containerID="cri-o://538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a" gracePeriod=30 Nov 03 14:46:37.729934 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:37.729900 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-jcn8k"] Nov 03 14:46:37.730387 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:37.730084 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" podUID="93952091-e0db-4779-9d1a-fab69c52f73f" containerName="manager" containerID="cri-o://b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8" gracePeriod=30 Nov 03 14:46:37.982886 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:37.982825 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:46:38.139652 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.139595 6389 generic.go:356] "Generic (PLEG): container finished" podID="93952091-e0db-4779-9d1a-fab69c52f73f" containerID="b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8" exitCode=0 Nov 03 14:46:38.139866 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.139666 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" Nov 03 14:46:38.139866 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.139684 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" event={"ID":"93952091-e0db-4779-9d1a-fab69c52f73f","Type":"ContainerDied","Data":"b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8"} Nov 03 14:46:38.139866 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.139764 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dfdc587c5-jcn8k" event={"ID":"93952091-e0db-4779-9d1a-fab69c52f73f","Type":"ContainerDied","Data":"129b9268cc77f245f751ee75dfe1509f29cdb5159afa5ffef95c235e4c75b3b8"} Nov 03 14:46:38.139866 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.139788 6389 scope.go:122] "RemoveContainer" containerID="b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8" Nov 03 14:46:38.149290 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.149261 6389 scope.go:122] "RemoveContainer" containerID="b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8" Nov 03 14:46:38.149553 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:46:38.149534 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8\": container with ID starting with b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8 not found: ID does not exist" containerID="b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8" Nov 03 14:46:38.149620 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.149563 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8"} err="failed to get container status \"b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8\": rpc error: code = NotFound desc = could not find container \"b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8\": container with ID starting with b08421ce34b5e26f848c221f04d75d060a18220e21afaec2794e30c00e4310f8 not found: ID does not exist" Nov 03 14:46:38.158132 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.158111 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93952091-e0db-4779-9d1a-fab69c52f73f-kube-api-access-7wdvk\" (UniqueName: \"kubernetes.io/projected/93952091-e0db-4779-9d1a-fab69c52f73f-kube-api-access-7wdvk\") pod \"93952091-e0db-4779-9d1a-fab69c52f73f\" (UID: \"93952091-e0db-4779-9d1a-fab69c52f73f\") " Nov 03 14:46:38.158231 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.158214 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-metrics-certs\") pod \"93952091-e0db-4779-9d1a-fab69c52f73f\" (UID: \"93952091-e0db-4779-9d1a-fab69c52f73f\") " Nov 03 14:46:38.158284 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.158246 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-webhook-cert\") pod \"93952091-e0db-4779-9d1a-fab69c52f73f\" (UID: \"93952091-e0db-4779-9d1a-fab69c52f73f\") " Nov 03 14:46:38.160307 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.160283 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-metrics-certs" pod "93952091-e0db-4779-9d1a-fab69c52f73f" (UID: "93952091-e0db-4779-9d1a-fab69c52f73f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:46:38.160394 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.160352 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-webhook-cert" pod "93952091-e0db-4779-9d1a-fab69c52f73f" (UID: "93952091-e0db-4779-9d1a-fab69c52f73f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:46:38.160394 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.160351 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93952091-e0db-4779-9d1a-fab69c52f73f-kube-api-access-7wdvk" pod "93952091-e0db-4779-9d1a-fab69c52f73f" (UID: "93952091-e0db-4779-9d1a-fab69c52f73f"). InnerVolumeSpecName "kube-api-access-7wdvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:46:38.259972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.259888 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wdvk\" (UniqueName: \"kubernetes.io/projected/93952091-e0db-4779-9d1a-fab69c52f73f-kube-api-access-7wdvk\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:38.259972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.259920 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:38.259972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.259934 6389 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93952091-e0db-4779-9d1a-fab69c52f73f-webhook-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:38.426557 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.426517 6389 patch_prober.go:29] interesting pod/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 14:46:38.426756 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.426593 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:46:38.487750 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.487723 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-jcn8k"] Nov 03 14:46:38.500234 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.500203 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-jcn8k"] Nov 03 14:46:38.617980 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:38.617910 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93952091-e0db-4779-9d1a-fab69c52f73f" path="/var/lib/kubelet/pods/93952091-e0db-4779-9d1a-fab69c52f73f/volumes" Nov 03 14:46:39.036470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.036444 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:46:39.147418 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.147329 6389 generic.go:356] "Generic (PLEG): container finished" podID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerID="538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a" exitCode=0 Nov 03 14:46:39.147418 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.147380 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" Nov 03 14:46:39.147714 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.147410 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" event={"ID":"873eaa3a-cbd6-4b40-881a-49d009f2f4f4","Type":"ContainerDied","Data":"538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a"} Nov 03 14:46:39.147714 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.147464 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b" event={"ID":"873eaa3a-cbd6-4b40-881a-49d009f2f4f4","Type":"ContainerDied","Data":"efdeee560b24ebb5bc31d65385594a36d045eb02de3a8f0378e29fe6b7ad98b9"} Nov 03 14:46:39.147714 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.147493 6389 scope.go:122] "RemoveContainer" containerID="538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a" Nov 03 14:46:39.157088 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.157071 6389 scope.go:122] "RemoveContainer" containerID="c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185" Nov 03 14:46:39.165267 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.165242 6389 scope.go:122] "RemoveContainer" containerID="538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a" Nov 03 14:46:39.165529 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:46:39.165510 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a\": container with ID starting with 538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a not found: ID does not exist" containerID="538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a" Nov 03 14:46:39.165746 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.165544 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a"} err="failed to get container status \"538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a\": rpc error: code = NotFound desc = could not find container \"538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a\": container with ID starting with 538271716b14661c4b7ce1b99a7039e710ab5a4bba0d3d1ce70a309a18ab8d8a not found: ID does not exist" Nov 03 14:46:39.165746 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.165568 6389 scope.go:122] "RemoveContainer" containerID="c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185" Nov 03 14:46:39.165973 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:46:39.165952 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185\": container with ID starting with c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185 not found: ID does not exist" containerID="c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185" Nov 03 14:46:39.166027 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.165984 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185"} err="failed to get container status \"c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185\": rpc error: code = NotFound desc = could not find container \"c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185\": container with ID starting with c56d4d77cafcaa2448e809b7276276610ebe4be5b0804e65204e29719e96d185 not found: ID does not exist" Nov 03 14:46:39.168274 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.168251 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-data\") pod \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " Nov 03 14:46:39.168356 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.168310 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-credential-socket\") pod \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " Nov 03 14:46:39.168418 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.168361 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-kube-api-access-zwjmg\" (UniqueName: \"kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-kube-api-access-zwjmg\") pod \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " Nov 03 14:46:39.168418 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.168409 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-podinfo\") pod \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " Nov 03 14:46:39.168534 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.168445 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-envoy\") pod \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " Nov 03 14:46:39.168534 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.168485 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-socket\") pod \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " Nov 03 14:46:39.168534 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.168526 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-certs\") pod \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " Nov 03 14:46:39.168718 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.168598 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-token\" (UniqueName: \"kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-token\") pod \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " Nov 03 14:46:39.168718 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.168691 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istiod-ca-cert\") pod \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\" (UID: \"873eaa3a-cbd6-4b40-881a-49d009f2f4f4\") " Nov 03 14:46:39.168828 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.168709 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-credential-socket" pod "873eaa3a-cbd6-4b40-881a-49d009f2f4f4" (UID: "873eaa3a-cbd6-4b40-881a-49d009f2f4f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:46:39.169150 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.168904 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-data" pod "873eaa3a-cbd6-4b40-881a-49d009f2f4f4" (UID: "873eaa3a-cbd6-4b40-881a-49d009f2f4f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:46:39.169150 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.168934 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-certs" pod "873eaa3a-cbd6-4b40-881a-49d009f2f4f4" (UID: "873eaa3a-cbd6-4b40-881a-49d009f2f4f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:46:39.169150 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.169069 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:39.169150 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.169089 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:39.169150 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.169105 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:39.169582 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.169325 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istiod-ca-cert" pod "873eaa3a-cbd6-4b40-881a-49d009f2f4f4" (UID: "873eaa3a-cbd6-4b40-881a-49d009f2f4f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:46:39.169670 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.169601 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-socket" pod "873eaa3a-cbd6-4b40-881a-49d009f2f4f4" (UID: "873eaa3a-cbd6-4b40-881a-49d009f2f4f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:46:39.170881 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.170850 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-kube-api-access-zwjmg" pod "873eaa3a-cbd6-4b40-881a-49d009f2f4f4" (UID: "873eaa3a-cbd6-4b40-881a-49d009f2f4f4"). InnerVolumeSpecName "kube-api-access-zwjmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:46:39.171296 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.171270 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-podinfo" pod "873eaa3a-cbd6-4b40-881a-49d009f2f4f4" (UID: "873eaa3a-cbd6-4b40-881a-49d009f2f4f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:46:39.171421 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.171400 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-envoy" pod "873eaa3a-cbd6-4b40-881a-49d009f2f4f4" (UID: "873eaa3a-cbd6-4b40-881a-49d009f2f4f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:46:39.171790 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.171757 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-token" pod "873eaa3a-cbd6-4b40-881a-49d009f2f4f4" (UID: "873eaa3a-cbd6-4b40-881a-49d009f2f4f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:46:39.242898 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.242849 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-jl224"] Nov 03 14:46:39.243497 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.243469 6389 cpu_manager.go:414] "containerMap: removing container" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" Nov 03 14:46:39.243497 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.243497 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" Nov 03 14:46:39.243732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.243522 6389 cpu_manager.go:414] "containerMap: removing container" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" Nov 03 14:46:39.243732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.243531 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" Nov 03 14:46:39.243732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.243553 6389 cpu_manager.go:414] "containerMap: removing container" podUID="93952091-e0db-4779-9d1a-fab69c52f73f" containerName="manager" Nov 03 14:46:39.243732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.243560 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93952091-e0db-4779-9d1a-fab69c52f73f" containerName="manager" Nov 03 14:46:39.243732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.243585 6389 cpu_manager.go:414] "containerMap: removing container" podUID="13cdb3ca-71a4-4a86-8bba-3c3ad895c12c" containerName="maas-api" Nov 03 14:46:39.243732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.243594 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13cdb3ca-71a4-4a86-8bba-3c3ad895c12c" containerName="maas-api" Nov 03 14:46:39.243732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.243701 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" Nov 03 14:46:39.243732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.243717 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="13cdb3ca-71a4-4a86-8bba-3c3ad895c12c" containerName="maas-api" Nov 03 14:46:39.243732 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.243727 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" containerName="istio-proxy" Nov 03 14:46:39.244220 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.243740 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="93952091-e0db-4779-9d1a-fab69c52f73f" containerName="manager" Nov 03 14:46:39.249101 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.249070 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:39.270448 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.270421 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:39.270448 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.270450 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwjmg\" (UniqueName: \"kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-kube-api-access-zwjmg\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:39.270765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.270464 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:39.270765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.270477 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:39.270765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.270488 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:39.270765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.270499 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/873eaa3a-cbd6-4b40-881a-49d009f2f4f4-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:46:39.270970 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.270875 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 14:46:39.271016 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.270997 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 14:46:39.271103 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.271088 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-62kg8\"" Nov 03 14:46:39.271208 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.271194 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 14:46:39.271331 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.271305 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-jl224"] Nov 03 14:46:39.371093 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.371057 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvd89\" (UniqueName: \"kubernetes.io/projected/5531b07e-f106-4c2f-bb93-575fb6723ed9-kube-api-access-mvd89\") pod \"maas-controller-6dfdc587c5-jl224\" (UID: \"5531b07e-f106-4c2f-bb93-575fb6723ed9\") " pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:39.371282 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.371114 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-metrics-certs\") pod \"maas-controller-6dfdc587c5-jl224\" (UID: \"5531b07e-f106-4c2f-bb93-575fb6723ed9\") " pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:39.371421 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.371392 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-webhook-cert\") pod \"maas-controller-6dfdc587c5-jl224\" (UID: \"5531b07e-f106-4c2f-bb93-575fb6723ed9\") " pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:39.472262 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.472230 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-metrics-certs\") pod \"maas-controller-6dfdc587c5-jl224\" (UID: \"5531b07e-f106-4c2f-bb93-575fb6723ed9\") " pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:39.472405 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.472346 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-webhook-cert\") pod \"maas-controller-6dfdc587c5-jl224\" (UID: \"5531b07e-f106-4c2f-bb93-575fb6723ed9\") " pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:39.472405 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.472373 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvd89\" (UniqueName: \"kubernetes.io/projected/5531b07e-f106-4c2f-bb93-575fb6723ed9-kube-api-access-mvd89\") pod \"maas-controller-6dfdc587c5-jl224\" (UID: \"5531b07e-f106-4c2f-bb93-575fb6723ed9\") " pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:39.475024 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.475001 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-webhook-cert\") pod \"maas-controller-6dfdc587c5-jl224\" (UID: \"5531b07e-f106-4c2f-bb93-575fb6723ed9\") " pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:39.475190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.475166 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-metrics-certs\") pod \"maas-controller-6dfdc587c5-jl224\" (UID: \"5531b07e-f106-4c2f-bb93-575fb6723ed9\") " pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:39.494896 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.494866 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvd89\" (UniqueName: \"kubernetes.io/projected/5531b07e-f106-4c2f-bb93-575fb6723ed9-kube-api-access-mvd89\") pod \"maas-controller-6dfdc587c5-jl224\" (UID: \"5531b07e-f106-4c2f-bb93-575fb6723ed9\") " pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:39.496435 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.496413 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b"] Nov 03 14:46:39.504435 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.504413 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8c02610e-openshift-default-5f957ccdc9-5sr9b"] Nov 03 14:46:39.561863 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.561830 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:39.879455 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:39.875984 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-jl224"] Nov 03 14:46:39.885072 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:46:39.885022 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5531b07e_f106_4c2f_bb93_575fb6723ed9.slice/crio-d833e1ebd8390f90b1cdc4e6fb0f64f9092991b0dc44e66623b5dc5655f59fee WatchSource:0}: Error finding container d833e1ebd8390f90b1cdc4e6fb0f64f9092991b0dc44e66623b5dc5655f59fee: Status 404 returned error can't find the container with id d833e1ebd8390f90b1cdc4e6fb0f64f9092991b0dc44e66623b5dc5655f59fee Nov 03 14:46:40.160312 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:40.160213 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dfdc587c5-jl224" event={"ID":"5531b07e-f106-4c2f-bb93-575fb6723ed9","Type":"ContainerStarted","Data":"d833e1ebd8390f90b1cdc4e6fb0f64f9092991b0dc44e66623b5dc5655f59fee"} Nov 03 14:46:40.616508 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:40.616467 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="873eaa3a-cbd6-4b40-881a-49d009f2f4f4" path="/var/lib/kubelet/pods/873eaa3a-cbd6-4b40-881a-49d009f2f4f4/volumes" Nov 03 14:46:41.170468 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:41.170417 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dfdc587c5-jl224" event={"ID":"5531b07e-f106-4c2f-bb93-575fb6723ed9","Type":"ContainerStarted","Data":"d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9"} Nov 03 14:46:41.171046 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:41.171024 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:41.217878 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:41.217807 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6dfdc587c5-jl224" podStartSLOduration=1.763654039 podStartE2EDuration="2.217786781s" totalImagesPullingTime="454.132742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:46:41.215860222 +0000 UTC m=+2076.774496705" watchObservedRunningTime="2026-11-03 14:46:41.217786781 +0000 UTC m=+2076.776423314" Nov 03 14:46:52.177787 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:46:52.177698 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:46:58.373908 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:46:58.373864 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:47:08.318961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.318926 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs"] Nov 03 14:47:08.323956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.323936 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.350177 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.350146 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs"] Nov 03 14:47:08.457423 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.457381 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.457649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.457466 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.457649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.457499 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.457649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.457527 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.457649 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.457551 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.457879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.457659 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.457879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.457696 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.457879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.457720 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.457879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.457862 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vsc\" (UniqueName: \"kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-kube-api-access-n2vsc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.558470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.558436 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.558674 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.558483 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.558761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.558705 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.558761 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.558736 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.558877 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.558767 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.558877 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.558809 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vsc\" (UniqueName: \"kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-kube-api-access-n2vsc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.558984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.558902 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.558984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.558937 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.558984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.558966 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.559288 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.559261 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.559535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.559279 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.559535 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.559356 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.559678 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.559661 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.559740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.559703 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.561289 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.561264 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.561390 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.561300 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.580853 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.580798 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vsc\" (UniqueName: \"kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-kube-api-access-n2vsc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.580853 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.580801 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5btvs\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.638709 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.638676 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:08.957524 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.957499 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs"] Nov 03 14:47:08.959565 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:47:08.959538 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c9a0b5_fad1_47ea_b811_168742c6fc6e.slice/crio-b4cbfa819d672db6e3e99e65651a1198a6d93e54c90fea3c09a30ccfb3a3cfe4 WatchSource:0}: Error finding container b4cbfa819d672db6e3e99e65651a1198a6d93e54c90fea3c09a30ccfb3a3cfe4: Status 404 returned error can't find the container with id b4cbfa819d672db6e3e99e65651a1198a6d93e54c90fea3c09a30ccfb3a3cfe4 Nov 03 14:47:08.961833 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.961801 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:47:08.961946 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.961882 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:47:08.961946 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:08.961928 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:47:09.301879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:09.301799 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" event={"ID":"26c9a0b5-fad1-47ea-b811-168742c6fc6e","Type":"ContainerStarted","Data":"aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424"} Nov 03 14:47:09.301879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:09.301857 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" event={"ID":"26c9a0b5-fad1-47ea-b811-168742c6fc6e","Type":"ContainerStarted","Data":"b4cbfa819d672db6e3e99e65651a1198a6d93e54c90fea3c09a30ccfb3a3cfe4"} Nov 03 14:47:09.346899 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:09.346841 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" podStartSLOduration=1.346827164 podStartE2EDuration="1.346827164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:47:09.345853067 +0000 UTC m=+2104.904489575" watchObservedRunningTime="2026-11-03 14:47:09.346827164 +0000 UTC m=+2104.905463648" Nov 03 14:47:09.645704 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:09.642563 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:10.644802 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:10.644765 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5btvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 14:47:10.645258 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:10.644830 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 14:47:10.847343 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:10.847311 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-jl224"] Nov 03 14:47:10.847580 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:10.847532 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6dfdc587c5-jl224" podUID="5531b07e-f106-4c2f-bb93-575fb6723ed9" containerName="manager" containerID="cri-o://d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9" gracePeriod=30 Nov 03 14:47:11.145028 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.144991 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:47:11.282428 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.282346 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-webhook-cert\") pod \"5531b07e-f106-4c2f-bb93-575fb6723ed9\" (UID: \"5531b07e-f106-4c2f-bb93-575fb6723ed9\") " Nov 03 14:47:11.282574 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.282463 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-metrics-certs\") pod \"5531b07e-f106-4c2f-bb93-575fb6723ed9\" (UID: \"5531b07e-f106-4c2f-bb93-575fb6723ed9\") " Nov 03 14:47:11.282574 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.282485 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5531b07e-f106-4c2f-bb93-575fb6723ed9-kube-api-access-mvd89\" (UniqueName: \"kubernetes.io/projected/5531b07e-f106-4c2f-bb93-575fb6723ed9-kube-api-access-mvd89\") pod \"5531b07e-f106-4c2f-bb93-575fb6723ed9\" (UID: \"5531b07e-f106-4c2f-bb93-575fb6723ed9\") " Nov 03 14:47:11.284647 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.284591 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-metrics-certs" pod "5531b07e-f106-4c2f-bb93-575fb6723ed9" (UID: "5531b07e-f106-4c2f-bb93-575fb6723ed9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:47:11.284844 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.284820 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-webhook-cert" pod "5531b07e-f106-4c2f-bb93-575fb6723ed9" (UID: "5531b07e-f106-4c2f-bb93-575fb6723ed9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:47:11.284927 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.284819 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5531b07e-f106-4c2f-bb93-575fb6723ed9-kube-api-access-mvd89" pod "5531b07e-f106-4c2f-bb93-575fb6723ed9" (UID: "5531b07e-f106-4c2f-bb93-575fb6723ed9"). InnerVolumeSpecName "kube-api-access-mvd89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:47:11.312518 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.312482 6389 generic.go:356] "Generic (PLEG): container finished" podID="5531b07e-f106-4c2f-bb93-575fb6723ed9" containerID="d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9" exitCode=0 Nov 03 14:47:11.312693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.312548 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dfdc587c5-jl224" Nov 03 14:47:11.312693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.312569 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dfdc587c5-jl224" event={"ID":"5531b07e-f106-4c2f-bb93-575fb6723ed9","Type":"ContainerDied","Data":"d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9"} Nov 03 14:47:11.312693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.312670 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dfdc587c5-jl224" event={"ID":"5531b07e-f106-4c2f-bb93-575fb6723ed9","Type":"ContainerDied","Data":"d833e1ebd8390f90b1cdc4e6fb0f64f9092991b0dc44e66623b5dc5655f59fee"} Nov 03 14:47:11.312809 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.312697 6389 scope.go:122] "RemoveContainer" containerID="d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9" Nov 03 14:47:11.346587 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.346564 6389 scope.go:122] "RemoveContainer" containerID="d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9" Nov 03 14:47:11.346926 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:47:11.346905 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9\": container with ID starting with d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9 not found: ID does not exist" containerID="d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9" Nov 03 14:47:11.346975 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.346936 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9"} err="failed to get container status \"d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9\": rpc error: code = NotFound desc = could not find container \"d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9\": container with ID starting with d2614eb991547365d437a4e5a3ecf35ee0f1c8868425988ca0df5edda6708ac9 not found: ID does not exist" Nov 03 14:47:11.383517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.383491 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvd89\" (UniqueName: \"kubernetes.io/projected/5531b07e-f106-4c2f-bb93-575fb6723ed9-kube-api-access-mvd89\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:47:11.383517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.383516 6389 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-webhook-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:47:11.383792 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.383525 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5531b07e-f106-4c2f-bb93-575fb6723ed9-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:47:11.384794 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.384771 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-jl224"] Nov 03 14:47:11.392542 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.392520 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-jl224"] Nov 03 14:47:11.641694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.641145 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5btvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 14:47:11.641694 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:11.641225 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 14:47:12.618134 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.618103 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5531b07e-f106-4c2f-bb93-575fb6723ed9" path="/var/lib/kubelet/pods/5531b07e-f106-4c2f-bb93-575fb6723ed9/volumes" Nov 03 14:47:12.643767 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.643726 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5btvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 14:47:12.643968 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.643806 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 14:47:12.743078 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.743039 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-vbgqs"] Nov 03 14:47:12.743947 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.743919 6389 cpu_manager.go:414] "containerMap: removing container" podUID="5531b07e-f106-4c2f-bb93-575fb6723ed9" containerName="manager" Nov 03 14:47:12.744520 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.744494 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5531b07e-f106-4c2f-bb93-575fb6723ed9" containerName="manager" Nov 03 14:47:12.744750 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.744733 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="5531b07e-f106-4c2f-bb93-575fb6723ed9" containerName="manager" Nov 03 14:47:12.749407 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.749386 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:12.771109 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.771079 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-vbgqs"] Nov 03 14:47:12.782089 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.782061 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 14:47:12.782224 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.782104 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-62kg8\"" Nov 03 14:47:12.782224 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.782113 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 14:47:12.782224 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.782220 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 14:47:12.796778 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.796752 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4csr\" (UniqueName: \"kubernetes.io/projected/fc2d0b86-48fd-48f6-89fa-150de7e46ca9-kube-api-access-d4csr\") pod \"maas-controller-6dfdc587c5-vbgqs\" (UID: \"fc2d0b86-48fd-48f6-89fa-150de7e46ca9\") " pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:12.796926 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.796912 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc2d0b86-48fd-48f6-89fa-150de7e46ca9-metrics-certs\") pod \"maas-controller-6dfdc587c5-vbgqs\" (UID: \"fc2d0b86-48fd-48f6-89fa-150de7e46ca9\") " pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:12.797071 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.797046 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2d0b86-48fd-48f6-89fa-150de7e46ca9-webhook-cert\") pod \"maas-controller-6dfdc587c5-vbgqs\" (UID: \"fc2d0b86-48fd-48f6-89fa-150de7e46ca9\") " pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:12.898431 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.898349 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2d0b86-48fd-48f6-89fa-150de7e46ca9-webhook-cert\") pod \"maas-controller-6dfdc587c5-vbgqs\" (UID: \"fc2d0b86-48fd-48f6-89fa-150de7e46ca9\") " pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:12.898431 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.898389 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4csr\" (UniqueName: \"kubernetes.io/projected/fc2d0b86-48fd-48f6-89fa-150de7e46ca9-kube-api-access-d4csr\") pod \"maas-controller-6dfdc587c5-vbgqs\" (UID: \"fc2d0b86-48fd-48f6-89fa-150de7e46ca9\") " pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:12.898646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.898445 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc2d0b86-48fd-48f6-89fa-150de7e46ca9-metrics-certs\") pod \"maas-controller-6dfdc587c5-vbgqs\" (UID: \"fc2d0b86-48fd-48f6-89fa-150de7e46ca9\") " pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:12.900836 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.900804 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2d0b86-48fd-48f6-89fa-150de7e46ca9-webhook-cert\") pod \"maas-controller-6dfdc587c5-vbgqs\" (UID: \"fc2d0b86-48fd-48f6-89fa-150de7e46ca9\") " pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:12.900836 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.900822 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc2d0b86-48fd-48f6-89fa-150de7e46ca9-metrics-certs\") pod \"maas-controller-6dfdc587c5-vbgqs\" (UID: \"fc2d0b86-48fd-48f6-89fa-150de7e46ca9\") " pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:12.944667 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:12.944593 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4csr\" (UniqueName: \"kubernetes.io/projected/fc2d0b86-48fd-48f6-89fa-150de7e46ca9-kube-api-access-d4csr\") pod \"maas-controller-6dfdc587c5-vbgqs\" (UID: \"fc2d0b86-48fd-48f6-89fa-150de7e46ca9\") " pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:13.062828 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:13.062793 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:13.375821 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:13.375796 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dfdc587c5-vbgqs"] Nov 03 14:47:13.378228 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:47:13.378200 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2d0b86_48fd_48f6_89fa_150de7e46ca9.slice/crio-fe83e5d0c535ac78022480796cb849334bc48cf0721d5b4f5ea470e1d02e51b7 WatchSource:0}: Error finding container fe83e5d0c535ac78022480796cb849334bc48cf0721d5b4f5ea470e1d02e51b7: Status 404 returned error can't find the container with id fe83e5d0c535ac78022480796cb849334bc48cf0721d5b4f5ea470e1d02e51b7 Nov 03 14:47:13.643681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:13.642937 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5btvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 14:47:13.643681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:13.642994 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 14:47:14.353447 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:14.353407 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" event={"ID":"fc2d0b86-48fd-48f6-89fa-150de7e46ca9","Type":"ContainerStarted","Data":"e49e143a8db6d5e27276b22107320baee21a97831b92d0bc141c08fbe9819ca3"} Nov 03 14:47:14.353655 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:14.353469 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:14.353655 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:14.353490 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" event={"ID":"fc2d0b86-48fd-48f6-89fa-150de7e46ca9","Type":"ContainerStarted","Data":"fe83e5d0c535ac78022480796cb849334bc48cf0721d5b4f5ea470e1d02e51b7"} Nov 03 14:47:14.399217 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:14.399162 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" podStartSLOduration=2.108052469 podStartE2EDuration="2.399138082s" totalImagesPullingTime="291.085613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:47:14.397965278 +0000 UTC m=+2109.956601763" watchObservedRunningTime="2026-11-03 14:47:14.399138082 +0000 UTC m=+2109.957774565" Nov 03 14:47:14.645382 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:14.645289 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5btvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 14:47:14.646015 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:14.645983 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 14:47:15.639434 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:15.639396 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5btvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 14:47:15.639646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:15.639448 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 14:47:16.639200 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:16.639158 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5btvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 03 14:47:16.639640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:16.639243 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 03 14:47:18.172390 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:18.172342 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:18.172923 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:18.172648 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:18.173993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:18.173972 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 14:47:25.360815 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:47:25.360782 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dfdc587c5-vbgqs" Nov 03 14:47:28.373462 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:47:28.373421 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:47:58.374670 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:47:58.374630 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:48:28.413529 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:48:28.413487 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:48:29.816772 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:48:29.816726 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:48:29.817284 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:48:29.816790 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:48:29.950807 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:48:29.950751 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:48:29.951191 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:48:29.951136 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:48:58.412560 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:48:58.412522 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:49:28.373977 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:49:28.373937 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:49:44.817841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:49:44.817748 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:49:44.817841 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:49:44.817806 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:49:44.946762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:49:44.946717 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:49:44.947064 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:49:44.947031 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:49:58.373687 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:49:58.373653 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:50:18.211188 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.211146 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql"] Nov 03 14:50:18.214460 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.214434 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.233737 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.233716 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-a58e3e-openshift-default-dockercfg-95tw6\"" Nov 03 14:50:18.247770 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.247749 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql"] Nov 03 14:50:18.320247 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.320208 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-istio-token\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.320444 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.320255 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-envoy\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.320444 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.320340 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-credential-socket\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.320444 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.320407 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-socket\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.320643 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.320470 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnbp\" (UniqueName: \"kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-kube-api-access-sjnbp\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.320643 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.320522 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd70ba0-113b-4458-afc4-5816a374036c-istiod-ca-cert\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.320753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.320638 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd70ba0-113b-4458-afc4-5816a374036c-istio-podinfo\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.320753 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.320691 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-data\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.320822 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.320753 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-certs\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.421374 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.421337 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd70ba0-113b-4458-afc4-5816a374036c-istiod-ca-cert\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.421583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.421393 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd70ba0-113b-4458-afc4-5816a374036c-istio-podinfo\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.421583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.421437 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-data\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.421583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.421479 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-certs\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.421583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.421516 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-istio-token\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.421833 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.421651 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-envoy\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.421833 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.421697 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-credential-socket\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.421833 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.421770 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-socket\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.421833 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.421815 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnbp\" (UniqueName: \"kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-kube-api-access-sjnbp\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.422052 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.421923 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-data\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.422180 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.422149 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-certs\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.422304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.422210 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-socket\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.422304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.422155 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-credential-socket\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.422304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.422221 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd70ba0-113b-4458-afc4-5816a374036c-istiod-ca-cert\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.424287 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.424264 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd70ba0-113b-4458-afc4-5816a374036c-istio-podinfo\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.424466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.424445 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-envoy\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.445023 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.444994 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnbp\" (UniqueName: \"kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-kube-api-access-sjnbp\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.445023 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.445017 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-istio-token\") pod \"e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.527676 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.527561 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:18.844705 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.844672 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql"] Nov 03 14:50:18.846464 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:50:18.846434 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd70ba0_113b_4458_afc4_5816a374036c.slice/crio-e137f204f982fd8957700471d5457ddb8e36b3c6f6390010604d66dd8916553b WatchSource:0}: Error finding container e137f204f982fd8957700471d5457ddb8e36b3c6f6390010604d66dd8916553b: Status 404 returned error can't find the container with id e137f204f982fd8957700471d5457ddb8e36b3c6f6390010604d66dd8916553b Nov 03 14:50:18.848868 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.848836 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:50:18.848960 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.848912 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:50:18.848960 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:18.848943 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:50:19.432362 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:19.432327 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" event={"ID":"5cd70ba0-113b-4458-afc4-5816a374036c","Type":"ContainerStarted","Data":"0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13"} Nov 03 14:50:19.432819 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:19.432410 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" event={"ID":"5cd70ba0-113b-4458-afc4-5816a374036c","Type":"ContainerStarted","Data":"e137f204f982fd8957700471d5457ddb8e36b3c6f6390010604d66dd8916553b"} Nov 03 14:50:19.506743 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:19.506689 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" podStartSLOduration=1.506675262 podStartE2EDuration="1.506675262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:50:19.493818342 +0000 UTC m=+2295.052454902" watchObservedRunningTime="2026-11-03 14:50:19.506675262 +0000 UTC m=+2295.065311747" Nov 03 14:50:19.528426 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:19.528391 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:19.533342 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:19.533314 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:20.437915 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:20.437877 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:20.439037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:20.439018 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:23.663585 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.663555 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8"] Nov 03 14:50:23.671918 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.671884 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.693946 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.693899 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8"] Nov 03 14:50:23.703393 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.703361 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a58e3e-serving-cert\"" Nov 03 14:50:23.780434 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.780396 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.780630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.780540 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.780630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.780571 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmkq7\" (UniqueName: \"kubernetes.io/projected/252237f7-24a0-4e85-be81-3997b87202dd-kube-api-access-cmkq7\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.780630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.780591 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252237f7-24a0-4e85-be81-3997b87202dd-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.881479 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.881446 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.881479 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.881482 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmkq7\" (UniqueName: \"kubernetes.io/projected/252237f7-24a0-4e85-be81-3997b87202dd-kube-api-access-cmkq7\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.881743 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.881503 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252237f7-24a0-4e85-be81-3997b87202dd-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.881743 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.881572 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.882409 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.882385 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252237f7-24a0-4e85-be81-3997b87202dd-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.884365 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.884341 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.884971 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.884928 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.914675 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.914374 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmkq7\" (UniqueName: \"kubernetes.io/projected/252237f7-24a0-4e85-be81-3997b87202dd-kube-api-access-cmkq7\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:23.987812 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:23.987781 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:24.062408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.062356 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z"] Nov 03 14:50:24.066137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.066120 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" Nov 03 14:50:24.091773 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.091630 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a58e3e\"" Nov 03 14:50:24.092232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.092211 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a7a5aa60-dockercfg-fmtqk\"" Nov 03 14:50:24.093485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.093463 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z"] Nov 03 14:50:24.166203 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.166105 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq"] Nov 03 14:50:24.170895 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.170871 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" Nov 03 14:50:24.180107 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.180077 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq"] Nov 03 14:50:24.194767 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.193812 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hkv\" (UniqueName: \"kubernetes.io/projected/b1ace0ea-e586-4370-b16a-38175a09f6af-kube-api-access-t5hkv\") pod \"payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z\" (UID: \"b1ace0ea-e586-4370-b16a-38175a09f6af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" Nov 03 14:50:24.194767 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.193946 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ace0ea-e586-4370-b16a-38175a09f6af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z\" (UID: \"b1ace0ea-e586-4370-b16a-38175a09f6af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" Nov 03 14:50:24.295310 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.295268 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ace0ea-e586-4370-b16a-38175a09f6af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z\" (UID: \"b1ace0ea-e586-4370-b16a-38175a09f6af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" Nov 03 14:50:24.295495 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.295354 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq\" (UID: \"9e33708b-3c1b-42dc-8b9b-3da56af21d1d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" Nov 03 14:50:24.295495 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.295389 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hkv\" (UniqueName: \"kubernetes.io/projected/b1ace0ea-e586-4370-b16a-38175a09f6af-kube-api-access-t5hkv\") pod \"payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z\" (UID: \"b1ace0ea-e586-4370-b16a-38175a09f6af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" Nov 03 14:50:24.295495 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.295419 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p62x5\" (UniqueName: \"kubernetes.io/projected/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-kube-api-access-p62x5\") pod \"payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq\" (UID: \"9e33708b-3c1b-42dc-8b9b-3da56af21d1d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" Nov 03 14:50:24.296034 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.296014 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ace0ea-e586-4370-b16a-38175a09f6af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z\" (UID: \"b1ace0ea-e586-4370-b16a-38175a09f6af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" Nov 03 14:50:24.306918 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.306896 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8"] Nov 03 14:50:24.308481 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:50:24.308461 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252237f7_24a0_4e85_be81_3997b87202dd.slice/crio-2a53e2590992a6be322d62b7125cbaddf4c88f3174f94d24b51f10861b43da69 WatchSource:0}: Error finding container 2a53e2590992a6be322d62b7125cbaddf4c88f3174f94d24b51f10861b43da69: Status 404 returned error can't find the container with id 2a53e2590992a6be322d62b7125cbaddf4c88f3174f94d24b51f10861b43da69 Nov 03 14:50:24.319137 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.319118 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hkv\" (UniqueName: \"kubernetes.io/projected/b1ace0ea-e586-4370-b16a-38175a09f6af-kube-api-access-t5hkv\") pod \"payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z\" (UID: \"b1ace0ea-e586-4370-b16a-38175a09f6af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" Nov 03 14:50:24.381859 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.381821 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" Nov 03 14:50:24.396729 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.396702 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq\" (UID: \"9e33708b-3c1b-42dc-8b9b-3da56af21d1d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" Nov 03 14:50:24.396850 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.396750 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62x5\" (UniqueName: \"kubernetes.io/projected/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-kube-api-access-p62x5\") pod \"payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq\" (UID: \"9e33708b-3c1b-42dc-8b9b-3da56af21d1d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" Nov 03 14:50:24.397349 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.397328 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq\" (UID: \"9e33708b-3c1b-42dc-8b9b-3da56af21d1d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" Nov 03 14:50:24.424693 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.424661 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62x5\" (UniqueName: \"kubernetes.io/projected/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-kube-api-access-p62x5\") pod \"payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq\" (UID: \"9e33708b-3c1b-42dc-8b9b-3da56af21d1d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" Nov 03 14:50:24.459249 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.459206 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" event={"ID":"252237f7-24a0-4e85-be81-3997b87202dd","Type":"ContainerStarted","Data":"2a53e2590992a6be322d62b7125cbaddf4c88f3174f94d24b51f10861b43da69"} Nov 03 14:50:24.504506 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.504473 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" Nov 03 14:50:24.683403 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.683382 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z"] Nov 03 14:50:24.685232 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:50:24.685207 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ace0ea_e586_4370_b16a_38175a09f6af.slice/crio-004457fefb5177313764e95f702a8e5add0cd2d50b0eed5eb119938ed880c8af WatchSource:0}: Error finding container 004457fefb5177313764e95f702a8e5add0cd2d50b0eed5eb119938ed880c8af: Status 404 returned error can't find the container with id 004457fefb5177313764e95f702a8e5add0cd2d50b0eed5eb119938ed880c8af Nov 03 14:50:24.811675 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:24.811545 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq"] Nov 03 14:50:24.814182 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:50:24.814148 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e33708b_3c1b_42dc_8b9b_3da56af21d1d.slice/crio-6756a74828c169ea4d4411b15729329691b0e0b5370141912bcdbf10393486b7 WatchSource:0}: Error finding container 6756a74828c169ea4d4411b15729329691b0e0b5370141912bcdbf10393486b7: Status 404 returned error can't find the container with id 6756a74828c169ea4d4411b15729329691b0e0b5370141912bcdbf10393486b7 Nov 03 14:50:25.465762 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:25.465720 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" event={"ID":"b1ace0ea-e586-4370-b16a-38175a09f6af","Type":"ContainerStarted","Data":"de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a"} Nov 03 14:50:25.465974 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:25.465773 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" event={"ID":"b1ace0ea-e586-4370-b16a-38175a09f6af","Type":"ContainerStarted","Data":"004457fefb5177313764e95f702a8e5add0cd2d50b0eed5eb119938ed880c8af"} Nov 03 14:50:25.465974 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:25.465825 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" Nov 03 14:50:25.467400 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:25.467373 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" event={"ID":"9e33708b-3c1b-42dc-8b9b-3da56af21d1d","Type":"ContainerStarted","Data":"5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba"} Nov 03 14:50:25.467551 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:25.467420 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" event={"ID":"9e33708b-3c1b-42dc-8b9b-3da56af21d1d","Type":"ContainerStarted","Data":"6756a74828c169ea4d4411b15729329691b0e0b5370141912bcdbf10393486b7"} Nov 03 14:50:25.467551 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:25.467462 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" Nov 03 14:50:25.468855 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:25.468837 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" event={"ID":"252237f7-24a0-4e85-be81-3997b87202dd","Type":"ContainerStarted","Data":"fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3"} Nov 03 14:50:25.468971 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:25.468958 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:25.510234 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:25.510181 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" podStartSLOduration=1.510162137 podStartE2EDuration="1.510162137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:50:25.509432391 +0000 UTC m=+2301.068068886" watchObservedRunningTime="2026-11-03 14:50:25.510162137 +0000 UTC m=+2301.068798621" Nov 03 14:50:25.555216 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:25.555160 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" podStartSLOduration=1.555145744 podStartE2EDuration="1.555145744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:50:25.554766685 +0000 UTC m=+2301.113403173" watchObservedRunningTime="2026-11-03 14:50:25.555145744 +0000 UTC m=+2301.113782228" Nov 03 14:50:25.599987 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:25.599940 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" podStartSLOduration=2.188016176 podStartE2EDuration="2.599927055s" totalImagesPullingTime="411.910879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:50:25.599626857 +0000 UTC m=+2301.158263332" watchObservedRunningTime="2026-11-03 14:50:25.599927055 +0000 UTC m=+2301.158563538" Nov 03 14:50:28.374187 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:50:28.374146 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:50:31.479906 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:31.479876 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:50:36.477198 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:36.477170 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" Nov 03 14:50:36.477656 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:36.477240 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" Nov 03 14:50:36.478726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:36.478705 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:50:36.478798 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:36.478756 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:50:37.530430 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:37.530401 6389 generic.go:356] "Generic (PLEG): container finished" podID="5cd70ba0-113b-4458-afc4-5816a374036c" containerID="0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13" exitCode=0 Nov 03 14:50:37.530904 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:37.530478 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" event={"ID":"5cd70ba0-113b-4458-afc4-5816a374036c","Type":"ContainerDied","Data":"0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13"} Nov 03 14:50:37.530951 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:37.530917 6389 scope.go:122] "RemoveContainer" containerID="0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13" Nov 03 14:50:37.532134 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:37.532107 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:50:37.532199 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:37.532186 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:50:37.532237 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:37.532221 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 14:50:38.549417 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:38.549382 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" event={"ID":"5cd70ba0-113b-4458-afc4-5816a374036c","Type":"ContainerStarted","Data":"3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215"} Nov 03 14:50:39.540633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:39.535985 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:39.541009 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:39.540977 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 14:50:39.541229 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:39.541199 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 14:50:40.536640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:40.529026 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 14:50:40.536640 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:40.529107 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 14:50:41.536634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:41.536023 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 14:50:41.536634 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:41.536102 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 14:50:42.538637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:42.534995 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 14:50:42.538637 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:42.535058 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 14:50:43.528222 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:43.528187 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 14:50:43.528412 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:43.528242 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 14:50:44.531663 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:44.531630 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:44.532079 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:44.531938 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:44.532773 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:50:44.532756 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:50:58.374150 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:50:58.374114 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:51:28.374675 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:51:28.374634 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:51:31.225782 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:31.225745 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq"] Nov 03 14:51:31.285851 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:31.285815 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z"] Nov 03 14:51:32.852955 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:32.852913 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8"] Nov 03 14:51:32.853395 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:32.853183 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" podUID="252237f7-24a0-4e85-be81-3997b87202dd" containerName="maas-api" containerID="cri-o://fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3" gracePeriod=30 Nov 03 14:51:34.100863 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.100835 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:51:34.169516 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.169478 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252237f7-24a0-4e85-be81-3997b87202dd-kube-api-access-cmkq7\" (UniqueName: \"kubernetes.io/projected/252237f7-24a0-4e85-be81-3997b87202dd-kube-api-access-cmkq7\") pod \"252237f7-24a0-4e85-be81-3997b87202dd\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " Nov 03 14:51:34.169710 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.169542 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252237f7-24a0-4e85-be81-3997b87202dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252237f7-24a0-4e85-be81-3997b87202dd-openshift-service-ca\") pod \"252237f7-24a0-4e85-be81-3997b87202dd\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " Nov 03 14:51:34.169710 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.169692 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-metrics-certs\") pod \"252237f7-24a0-4e85-be81-3997b87202dd\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " Nov 03 14:51:34.169840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.169734 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-maas-api-tls\") pod \"252237f7-24a0-4e85-be81-3997b87202dd\" (UID: \"252237f7-24a0-4e85-be81-3997b87202dd\") " Nov 03 14:51:34.170037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.170000 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252237f7-24a0-4e85-be81-3997b87202dd-openshift-service-ca" pod "252237f7-24a0-4e85-be81-3997b87202dd" (UID: "252237f7-24a0-4e85-be81-3997b87202dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:51:34.171887 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.171862 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-metrics-certs" pod "252237f7-24a0-4e85-be81-3997b87202dd" (UID: "252237f7-24a0-4e85-be81-3997b87202dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:51:34.172049 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.172028 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-maas-api-tls" pod "252237f7-24a0-4e85-be81-3997b87202dd" (UID: "252237f7-24a0-4e85-be81-3997b87202dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:51:34.172336 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.172316 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252237f7-24a0-4e85-be81-3997b87202dd-kube-api-access-cmkq7" pod "252237f7-24a0-4e85-be81-3997b87202dd" (UID: "252237f7-24a0-4e85-be81-3997b87202dd"). InnerVolumeSpecName "kube-api-access-cmkq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:51:34.270840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.270756 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:51:34.270840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.270783 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252237f7-24a0-4e85-be81-3997b87202dd-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:51:34.270840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.270792 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmkq7\" (UniqueName: \"kubernetes.io/projected/252237f7-24a0-4e85-be81-3997b87202dd-kube-api-access-cmkq7\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:51:34.270840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.270801 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252237f7-24a0-4e85-be81-3997b87202dd-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:51:34.856310 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.856279 6389 generic.go:356] "Generic (PLEG): container finished" podID="252237f7-24a0-4e85-be81-3997b87202dd" containerID="fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3" exitCode=0 Nov 03 14:51:34.856491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.856328 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" Nov 03 14:51:34.856491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.856369 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" event={"ID":"252237f7-24a0-4e85-be81-3997b87202dd","Type":"ContainerDied","Data":"fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3"} Nov 03 14:51:34.856491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.856428 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8" event={"ID":"252237f7-24a0-4e85-be81-3997b87202dd","Type":"ContainerDied","Data":"2a53e2590992a6be322d62b7125cbaddf4c88f3174f94d24b51f10861b43da69"} Nov 03 14:51:34.856491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.856449 6389 scope.go:122] "RemoveContainer" containerID="fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3" Nov 03 14:51:34.865957 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.865936 6389 scope.go:122] "RemoveContainer" containerID="fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3" Nov 03 14:51:34.866238 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:51:34.866220 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3\": container with ID starting with fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3 not found: ID does not exist" containerID="fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3" Nov 03 14:51:34.866303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.866250 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3"} err="failed to get container status \"fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3\": rpc error: code = NotFound desc = could not find container \"fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3\": container with ID starting with fe62ca86ee00fe8b73c7de7aaa2b9cbed6c4cc42766401cd52fe581b4cfceda3 not found: ID does not exist" Nov 03 14:51:34.916080 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.916003 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8"] Nov 03 14:51:34.928675 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:34.928650 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-jmjx8"] Nov 03 14:51:36.226526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:36.226466 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" podUID="9e33708b-3c1b-42dc-8b9b-3da56af21d1d" containerName="payload-processing" containerID="cri-o://5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba" gracePeriod=25 Nov 03 14:51:36.287111 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:36.287066 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" podUID="b1ace0ea-e586-4370-b16a-38175a09f6af" containerName="payload-pre-processing" containerID="cri-o://de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a" gracePeriod=25 Nov 03 14:51:36.475349 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:36.475308 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" podUID="b1ace0ea-e586-4370-b16a-38175a09f6af" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:51:36.475349 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:36.475328 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" podUID="9e33708b-3c1b-42dc-8b9b-3da56af21d1d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:51:36.616445 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:36.616360 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252237f7-24a0-4e85-be81-3997b87202dd" path="/var/lib/kubelet/pods/252237f7-24a0-4e85-be81-3997b87202dd/volumes" Nov 03 14:51:46.475919 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:46.475880 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" podUID="b1ace0ea-e586-4370-b16a-38175a09f6af" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:51:46.475919 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:46.475902 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" podUID="9e33708b-3c1b-42dc-8b9b-3da56af21d1d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:51:56.474361 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:51:56.474331 6389 logging.go:55] [core] [Channel #3029 SubChannel #3031]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.145:50052", ServerName: "10.134.0.145:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.145:50052: connect: connection refused" Nov 03 14:51:56.550129 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.550100 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" Nov 03 14:51:56.573013 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.572982 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" Nov 03 14:51:56.694797 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.694766 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ace0ea-e586-4370-b16a-38175a09f6af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ace0ea-e586-4370-b16a-38175a09f6af-plugins-config-volume\") pod \"b1ace0ea-e586-4370-b16a-38175a09f6af\" (UID: \"b1ace0ea-e586-4370-b16a-38175a09f6af\") " Nov 03 14:51:56.694797 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.694801 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-plugins-config-volume\") pod \"9e33708b-3c1b-42dc-8b9b-3da56af21d1d\" (UID: \"9e33708b-3c1b-42dc-8b9b-3da56af21d1d\") " Nov 03 14:51:56.695026 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.694820 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-kube-api-access-p62x5\" (UniqueName: \"kubernetes.io/projected/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-kube-api-access-p62x5\") pod \"9e33708b-3c1b-42dc-8b9b-3da56af21d1d\" (UID: \"9e33708b-3c1b-42dc-8b9b-3da56af21d1d\") " Nov 03 14:51:56.695026 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.694867 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ace0ea-e586-4370-b16a-38175a09f6af-kube-api-access-t5hkv\" (UniqueName: \"kubernetes.io/projected/b1ace0ea-e586-4370-b16a-38175a09f6af-kube-api-access-t5hkv\") pod \"b1ace0ea-e586-4370-b16a-38175a09f6af\" (UID: \"b1ace0ea-e586-4370-b16a-38175a09f6af\") " Nov 03 14:51:56.695179 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.695155 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ace0ea-e586-4370-b16a-38175a09f6af-plugins-config-volume" pod "b1ace0ea-e586-4370-b16a-38175a09f6af" (UID: "b1ace0ea-e586-4370-b16a-38175a09f6af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:51:56.695258 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.695197 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-plugins-config-volume" pod "9e33708b-3c1b-42dc-8b9b-3da56af21d1d" (UID: "9e33708b-3c1b-42dc-8b9b-3da56af21d1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:51:56.697110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.697080 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-kube-api-access-p62x5" pod "9e33708b-3c1b-42dc-8b9b-3da56af21d1d" (UID: "9e33708b-3c1b-42dc-8b9b-3da56af21d1d"). InnerVolumeSpecName "kube-api-access-p62x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:51:56.697254 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.697225 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ace0ea-e586-4370-b16a-38175a09f6af-kube-api-access-t5hkv" pod "b1ace0ea-e586-4370-b16a-38175a09f6af" (UID: "b1ace0ea-e586-4370-b16a-38175a09f6af"). InnerVolumeSpecName "kube-api-access-t5hkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:51:56.796380 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.796304 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5hkv\" (UniqueName: \"kubernetes.io/projected/b1ace0ea-e586-4370-b16a-38175a09f6af-kube-api-access-t5hkv\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:51:56.796380 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.796332 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ace0ea-e586-4370-b16a-38175a09f6af-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:51:56.796380 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.796343 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:51:56.796380 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.796352 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p62x5\" (UniqueName: \"kubernetes.io/projected/9e33708b-3c1b-42dc-8b9b-3da56af21d1d-kube-api-access-p62x5\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:51:56.964113 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.964081 6389 generic.go:356] "Generic (PLEG): container finished" podID="b1ace0ea-e586-4370-b16a-38175a09f6af" containerID="de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a" exitCode=0 Nov 03 14:51:56.964312 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.964129 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" Nov 03 14:51:56.964312 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.964170 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" event={"ID":"b1ace0ea-e586-4370-b16a-38175a09f6af","Type":"ContainerDied","Data":"de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a"} Nov 03 14:51:56.964312 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.964230 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" event={"ID":"b1ace0ea-e586-4370-b16a-38175a09f6af","Type":"ContainerDied","Data":"004457fefb5177313764e95f702a8e5add0cd2d50b0eed5eb119938ed880c8af"} Nov 03 14:51:56.964312 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.964259 6389 scope.go:122] "RemoveContainer" containerID="de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a" Nov 03 14:51:56.965676 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.965653 6389 generic.go:356] "Generic (PLEG): container finished" podID="9e33708b-3c1b-42dc-8b9b-3da56af21d1d" containerID="5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba" exitCode=0 Nov 03 14:51:56.965804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.965678 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" event={"ID":"9e33708b-3c1b-42dc-8b9b-3da56af21d1d","Type":"ContainerDied","Data":"5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba"} Nov 03 14:51:56.965804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.965706 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" Nov 03 14:51:56.965804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.965727 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" event={"ID":"9e33708b-3c1b-42dc-8b9b-3da56af21d1d","Type":"ContainerDied","Data":"6756a74828c169ea4d4411b15729329691b0e0b5370141912bcdbf10393486b7"} Nov 03 14:51:56.973417 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.973400 6389 scope.go:122] "RemoveContainer" containerID="de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a" Nov 03 14:51:56.973701 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:51:56.973681 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a\": container with ID starting with de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a not found: ID does not exist" containerID="de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a" Nov 03 14:51:56.973802 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.973707 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a"} err="failed to get container status \"de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a\": rpc error: code = NotFound desc = could not find container \"de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a\": container with ID starting with de3c6c3c905f117c722962745e9f37a7bffb2f142278ced5f299a0cc6565f30a not found: ID does not exist" Nov 03 14:51:56.973802 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.973724 6389 scope.go:122] "RemoveContainer" containerID="5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba" Nov 03 14:51:56.981671 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.981654 6389 scope.go:122] "RemoveContainer" containerID="5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba" Nov 03 14:51:56.981937 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:51:56.981909 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba\": container with ID starting with 5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba not found: ID does not exist" containerID="5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba" Nov 03 14:51:56.981988 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:56.981935 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba"} err="failed to get container status \"5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba\": rpc error: code = NotFound desc = could not find container \"5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba\": container with ID starting with 5348f7d389717d5e69a325e6f76c7006666423d21c3e8303b07613c0029f33ba not found: ID does not exist" Nov 03 14:51:57.020955 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:57.020925 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z"] Nov 03 14:51:57.033471 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:57.033443 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z"] Nov 03 14:51:57.064961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:57.064893 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq"] Nov 03 14:51:57.071316 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:57.071295 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq"] Nov 03 14:51:57.474793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:57.474750 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5j5x6z" podUID="b1ace0ea-e586-4370-b16a-38175a09f6af" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.145:50052\" within 1s: context deadline exceeded" Nov 03 14:51:57.474793 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:57.474774 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cpvktq" podUID="9e33708b-3c1b-42dc-8b9b-3da56af21d1d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.146:50052\" within 1s: context deadline exceeded" Nov 03 14:51:57.475251 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:51:57.474850 6389 logging.go:55] [core] [Channel #3030 SubChannel #3032]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.146:50052", ServerName: "10.134.0.146:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.146:50052: operation was canceled" Nov 03 14:51:58.372984 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:51:58.372946 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:51:58.616866 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:58.616832 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e33708b-3c1b-42dc-8b9b-3da56af21d1d" path="/var/lib/kubelet/pods/9e33708b-3c1b-42dc-8b9b-3da56af21d1d/volumes" Nov 03 14:51:58.617391 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:51:58.617191 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ace0ea-e586-4370-b16a-38175a09f6af" path="/var/lib/kubelet/pods/b1ace0ea-e586-4370-b16a-38175a09f6af/volumes" Nov 03 14:52:05.121980 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.121951 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz"] Nov 03 14:52:05.122578 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.122557 6389 cpu_manager.go:414] "containerMap: removing container" podUID="252237f7-24a0-4e85-be81-3997b87202dd" containerName="maas-api" Nov 03 14:52:05.122686 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.122581 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252237f7-24a0-4e85-be81-3997b87202dd" containerName="maas-api" Nov 03 14:52:05.122686 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.122599 6389 cpu_manager.go:414] "containerMap: removing container" podUID="9e33708b-3c1b-42dc-8b9b-3da56af21d1d" containerName="payload-processing" Nov 03 14:52:05.122686 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.122623 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e33708b-3c1b-42dc-8b9b-3da56af21d1d" containerName="payload-processing" Nov 03 14:52:05.122686 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.122648 6389 cpu_manager.go:414] "containerMap: removing container" podUID="b1ace0ea-e586-4370-b16a-38175a09f6af" containerName="payload-pre-processing" Nov 03 14:52:05.122686 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.122656 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ace0ea-e586-4370-b16a-38175a09f6af" containerName="payload-pre-processing" Nov 03 14:52:05.122951 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.122761 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ace0ea-e586-4370-b16a-38175a09f6af" containerName="payload-pre-processing" Nov 03 14:52:05.122951 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.122777 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="252237f7-24a0-4e85-be81-3997b87202dd" containerName="maas-api" Nov 03 14:52:05.122951 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.122789 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e33708b-3c1b-42dc-8b9b-3da56af21d1d" containerName="payload-processing" Nov 03 14:52:05.126314 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.126291 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.163386 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.163359 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a58e3e-serving-cert\"" Nov 03 14:52:05.165960 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.165931 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz"] Nov 03 14:52:05.280868 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.280831 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.281052 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.280899 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.281052 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.280961 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv8xn\" (UniqueName: \"kubernetes.io/projected/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-kube-api-access-wv8xn\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.281147 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.281122 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.382449 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.382355 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.382449 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.382403 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.382449 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.382444 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.382778 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.382466 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv8xn\" (UniqueName: \"kubernetes.io/projected/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-kube-api-access-wv8xn\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.382778 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:52:05.382600 6389 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-serving-cert: secret "maas-api-e2e-worker-single-main-a58e3e-serving-cert" not found Nov 03 14:52:05.382778 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:52:05.382703 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-maas-api-tls podName:4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce nodeName:}" failed. No retries permitted until 2026-11-03 14:52:05.882676493 +0000 UTC m=+2401.441312969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-maas-api-tls") pod "maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" (UID: "4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce") : secret "maas-api-e2e-worker-single-main-a58e3e-serving-cert" not found Nov 03 14:52:05.383390 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.383364 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.385016 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.384986 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.404545 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.404518 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv8xn\" (UniqueName: \"kubernetes.io/projected/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-kube-api-access-wv8xn\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.890752 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.890715 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:05.893301 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:05.893277 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:06.038869 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:06.038830 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:06.343131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:06.343097 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz"] Nov 03 14:52:06.344426 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:52:06.344400 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aec69dc_44be_41e4_8eb2_5fa48cb6d4ce.slice/crio-953ca98951902429b3aad8c1d95cf3e0296b7713c016d5cd09a9a74f1b1fd2e8 WatchSource:0}: Error finding container 953ca98951902429b3aad8c1d95cf3e0296b7713c016d5cd09a9a74f1b1fd2e8: Status 404 returned error can't find the container with id 953ca98951902429b3aad8c1d95cf3e0296b7713c016d5cd09a9a74f1b1fd2e8 Nov 03 14:52:06.345970 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:06.345950 6389 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:52:07.019975 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:07.019854 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" event={"ID":"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce","Type":"ContainerStarted","Data":"9968eaafcd2e5dd5cd463c2f92c443677bd55658380b82d060e6b0d67d0381c1"} Nov 03 14:52:07.020142 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:07.019988 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:07.020142 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:07.020020 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" event={"ID":"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce","Type":"ContainerStarted","Data":"953ca98951902429b3aad8c1d95cf3e0296b7713c016d5cd09a9a74f1b1fd2e8"} Nov 03 14:52:07.065477 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:07.065428 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" podStartSLOduration=1.733453371 podStartE2EDuration="2.065415493s" totalImagesPullingTime="331.962122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:52:07.064666653 +0000 UTC m=+2402.623303138" watchObservedRunningTime="2026-11-03 14:52:07.065415493 +0000 UTC m=+2402.624051977" Nov 03 14:52:13.032107 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:13.032079 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:28.373458 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:52:28.373422 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:52:28.903954 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:28.903914 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz"] Nov 03 14:52:28.904226 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:28.904173 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" podUID="4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce" containerName="maas-api" containerID="cri-o://9968eaafcd2e5dd5cd463c2f92c443677bd55658380b82d060e6b0d67d0381c1" gracePeriod=30 Nov 03 14:52:29.151057 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.151015 6389 generic.go:356] "Generic (PLEG): container finished" podID="4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce" containerID="9968eaafcd2e5dd5cd463c2f92c443677bd55658380b82d060e6b0d67d0381c1" exitCode=0 Nov 03 14:52:29.151333 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.151150 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" event={"ID":"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce","Type":"ContainerDied","Data":"9968eaafcd2e5dd5cd463c2f92c443677bd55658380b82d060e6b0d67d0381c1"} Nov 03 14:52:29.197907 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.197880 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:29.325225 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.325196 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-openshift-service-ca\") pod \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " Nov 03 14:52:29.325420 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.325286 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-metrics-certs\") pod \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " Nov 03 14:52:29.325502 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.325421 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-maas-api-tls\") pod \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " Nov 03 14:52:29.325502 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.325484 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-kube-api-access-wv8xn\" (UniqueName: \"kubernetes.io/projected/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-kube-api-access-wv8xn\") pod \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\" (UID: \"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce\") " Nov 03 14:52:29.325622 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.325580 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-openshift-service-ca" pod "4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce" (UID: "4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:52:29.325840 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.325823 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:29.327549 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.327521 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-metrics-certs" pod "4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce" (UID: "4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:52:29.327685 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.327547 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-maas-api-tls" pod "4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce" (UID: "4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:52:29.327802 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.327786 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-kube-api-access-wv8xn" pod "4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce" (UID: "4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce"). InnerVolumeSpecName "kube-api-access-wv8xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:52:29.426414 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.426332 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:29.426414 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.426363 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:29.426414 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:29.426372 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv8xn\" (UniqueName: \"kubernetes.io/projected/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce-kube-api-access-wv8xn\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:30.157484 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:30.157453 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" Nov 03 14:52:30.157805 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:30.157517 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz" event={"ID":"4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce","Type":"ContainerDied","Data":"953ca98951902429b3aad8c1d95cf3e0296b7713c016d5cd09a9a74f1b1fd2e8"} Nov 03 14:52:30.157805 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:30.157587 6389 scope.go:122] "RemoveContainer" containerID="9968eaafcd2e5dd5cd463c2f92c443677bd55658380b82d060e6b0d67d0381c1" Nov 03 14:52:30.199975 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:30.199945 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql"] Nov 03 14:52:30.200192 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:30.200169 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" containerID="cri-o://3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215" gracePeriod=30 Nov 03 14:52:30.212530 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:30.212506 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz"] Nov 03 14:52:30.217880 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:30.217860 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a58e3e-6db6c8548c-h6rvz"] Nov 03 14:52:30.617622 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:30.617580 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce" path="/var/lib/kubelet/pods/4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce/volumes" Nov 03 14:52:35.458696 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.458670 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:52:35.590806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.590719 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cd70ba0-113b-4458-afc4-5816a374036c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd70ba0-113b-4458-afc4-5816a374036c-istio-podinfo\") pod \"5cd70ba0-113b-4458-afc4-5816a374036c\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " Nov 03 14:52:35.590806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.590778 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-credential-socket\") pod \"5cd70ba0-113b-4458-afc4-5816a374036c\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " Nov 03 14:52:35.590806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.590798 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-certs\") pod \"5cd70ba0-113b-4458-afc4-5816a374036c\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " Nov 03 14:52:35.591077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.590821 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-envoy\") pod \"5cd70ba0-113b-4458-afc4-5816a374036c\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " Nov 03 14:52:35.591077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.590938 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-data\") pod \"5cd70ba0-113b-4458-afc4-5816a374036c\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " Nov 03 14:52:35.591077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591012 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-istio-token\" (UniqueName: \"kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-istio-token\") pod \"5cd70ba0-113b-4458-afc4-5816a374036c\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " Nov 03 14:52:35.591077 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591052 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-kube-api-access-sjnbp\" (UniqueName: \"kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-kube-api-access-sjnbp\") pod \"5cd70ba0-113b-4458-afc4-5816a374036c\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " Nov 03 14:52:35.591316 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591106 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-socket\") pod \"5cd70ba0-113b-4458-afc4-5816a374036c\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " Nov 03 14:52:35.591316 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591125 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-credential-socket" pod "5cd70ba0-113b-4458-afc4-5816a374036c" (UID: "5cd70ba0-113b-4458-afc4-5816a374036c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:52:35.591316 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591140 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-certs" pod "5cd70ba0-113b-4458-afc4-5816a374036c" (UID: "5cd70ba0-113b-4458-afc4-5816a374036c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:52:35.591316 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591214 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-data" pod "5cd70ba0-113b-4458-afc4-5816a374036c" (UID: "5cd70ba0-113b-4458-afc4-5816a374036c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:52:35.591316 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591236 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd70ba0-113b-4458-afc4-5816a374036c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd70ba0-113b-4458-afc4-5816a374036c-istiod-ca-cert\") pod \"5cd70ba0-113b-4458-afc4-5816a374036c\" (UID: \"5cd70ba0-113b-4458-afc4-5816a374036c\") " Nov 03 14:52:35.591599 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591337 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-socket" pod "5cd70ba0-113b-4458-afc4-5816a374036c" (UID: "5cd70ba0-113b-4458-afc4-5816a374036c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:52:35.591681 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591590 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd70ba0-113b-4458-afc4-5816a374036c-istiod-ca-cert" pod "5cd70ba0-113b-4458-afc4-5816a374036c" (UID: "5cd70ba0-113b-4458-afc4-5816a374036c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:52:35.591879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591854 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:35.591979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591883 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:35.591979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591899 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:35.591979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591912 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:35.591979 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.591928 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd70ba0-113b-4458-afc4-5816a374036c-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:35.593210 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.593185 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cd70ba0-113b-4458-afc4-5816a374036c-istio-podinfo" pod "5cd70ba0-113b-4458-afc4-5816a374036c" (UID: "5cd70ba0-113b-4458-afc4-5816a374036c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:52:35.593311 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.593221 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-envoy" pod "5cd70ba0-113b-4458-afc4-5816a374036c" (UID: "5cd70ba0-113b-4458-afc4-5816a374036c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:52:35.593948 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.593925 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-kube-api-access-sjnbp" pod "5cd70ba0-113b-4458-afc4-5816a374036c" (UID: "5cd70ba0-113b-4458-afc4-5816a374036c"). InnerVolumeSpecName "kube-api-access-sjnbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:52:35.594036 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.593926 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-istio-token" pod "5cd70ba0-113b-4458-afc4-5816a374036c" (UID: "5cd70ba0-113b-4458-afc4-5816a374036c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:52:35.693368 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.693328 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd70ba0-113b-4458-afc4-5816a374036c-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:35.693368 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.693360 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd70ba0-113b-4458-afc4-5816a374036c-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:35.693368 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.693376 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:35.693631 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:35.693388 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjnbp\" (UniqueName: \"kubernetes.io/projected/5cd70ba0-113b-4458-afc4-5816a374036c-kube-api-access-sjnbp\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:52:36.188091 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.188058 6389 generic.go:356] "Generic (PLEG): container finished" podID="5cd70ba0-113b-4458-afc4-5816a374036c" containerID="3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215" exitCode=0 Nov 03 14:52:36.188371 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.188104 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" Nov 03 14:52:36.188371 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.188139 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" event={"ID":"5cd70ba0-113b-4458-afc4-5816a374036c","Type":"ContainerDied","Data":"3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215"} Nov 03 14:52:36.188371 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.188195 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql" event={"ID":"5cd70ba0-113b-4458-afc4-5816a374036c","Type":"ContainerDied","Data":"e137f204f982fd8957700471d5457ddb8e36b3c6f6390010604d66dd8916553b"} Nov 03 14:52:36.188371 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.188220 6389 scope.go:122] "RemoveContainer" containerID="3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215" Nov 03 14:52:36.197748 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.197729 6389 scope.go:122] "RemoveContainer" containerID="0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13" Nov 03 14:52:36.206378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.206357 6389 scope.go:122] "RemoveContainer" containerID="3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215" Nov 03 14:52:36.206657 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:52:36.206639 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215\": container with ID starting with 3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215 not found: ID does not exist" containerID="3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215" Nov 03 14:52:36.206710 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.206669 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215"} err="failed to get container status \"3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215\": rpc error: code = NotFound desc = could not find container \"3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215\": container with ID starting with 3e3be6646d3057cb7a531cea5d53498c9235d8cec3711a8e1c960b5bbe2e2215 not found: ID does not exist" Nov 03 14:52:36.206710 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.206691 6389 scope.go:122] "RemoveContainer" containerID="0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13" Nov 03 14:52:36.206973 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:52:36.206956 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13\": container with ID starting with 0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13 not found: ID does not exist" containerID="0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13" Nov 03 14:52:36.207015 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.206981 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13"} err="failed to get container status \"0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13\": rpc error: code = NotFound desc = could not find container \"0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13\": container with ID starting with 0d2d45da8feb1c3054fe3330093d750b30c9130bd59c51e3010194b7a7f8bc13 not found: ID does not exist" Nov 03 14:52:36.255559 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.255520 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql"] Nov 03 14:52:36.268006 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.267981 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a58e3e-openshift-default-67d65d499c9vvql"] Nov 03 14:52:36.616921 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:36.616844 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" path="/var/lib/kubelet/pods/5cd70ba0-113b-4458-afc4-5816a374036c/volumes" Nov 03 14:52:58.412282 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:52:58.412202 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:52:59.825765 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:59.825721 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:52:59.826454 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:59.826419 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:52:59.955646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:59.954730 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:52:59.955646 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:52:59.954799 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:53:28.373395 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:53:28.373357 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:53:34.203696 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:53:34.203643 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5btvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:53:34.204276 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:53:34.203726 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:53:44.818809 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:53:44.818767 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:53:44.819564 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:53:44.819527 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:53:44.953633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:53:44.952739 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:53:44.953633 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:53:44.952844 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:53:58.373596 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:53:58.373554 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:54:13.846083 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.846007 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6"] Nov 03 14:54:13.846711 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.846691 6389 cpu_manager.go:414] "containerMap: removing container" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" Nov 03 14:54:13.846771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.846715 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" Nov 03 14:54:13.846771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.846754 6389 cpu_manager.go:414] "containerMap: removing container" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" Nov 03 14:54:13.846771 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.846763 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" Nov 03 14:54:13.846909 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.846785 6389 cpu_manager.go:414] "containerMap: removing container" podUID="4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce" containerName="maas-api" Nov 03 14:54:13.846909 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.846793 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce" containerName="maas-api" Nov 03 14:54:13.846909 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.846897 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" Nov 03 14:54:13.846909 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.846907 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aec69dc-44be-41e4-8eb2-5fa48cb6d4ce" containerName="maas-api" Nov 03 14:54:13.850378 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.850358 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:13.880807 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.880780 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-72b663-serving-cert\"" Nov 03 14:54:13.883184 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.883154 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6"] Nov 03 14:54:13.941309 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.941282 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-metrics-certs\") pod \"maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:13.941481 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.941321 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:13.941481 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.941388 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/120469ce-b114-4f5f-b04e-cda3add1ffa2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:13.941481 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:13.941439 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs497\" (UniqueName: \"kubernetes.io/projected/120469ce-b114-4f5f-b04e-cda3add1ffa2-kube-api-access-rs497\") pod \"maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:14.042882 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:14.042845 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-metrics-certs\") pod \"maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:14.042882 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:14.042891 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:14.043128 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:14.042913 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/120469ce-b114-4f5f-b04e-cda3add1ffa2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:14.043128 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:14.042993 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs497\" (UniqueName: \"kubernetes.io/projected/120469ce-b114-4f5f-b04e-cda3add1ffa2-kube-api-access-rs497\") pod \"maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:14.043578 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:14.043536 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/120469ce-b114-4f5f-b04e-cda3add1ffa2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:14.045556 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:14.045531 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:14.045679 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:14.045587 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-metrics-certs\") pod \"maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:14.066503 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:14.066483 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs497\" (UniqueName: \"kubernetes.io/projected/120469ce-b114-4f5f-b04e-cda3add1ffa2-kube-api-access-rs497\") pod \"maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:14.174791 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:14.174763 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:14.472987 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:14.472960 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6"] Nov 03 14:54:14.475555 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:54:14.475526 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120469ce_b114_4f5f_b04e_cda3add1ffa2.slice/crio-c846d35166eec06c56b3b14746bc8da097f9becbb0a1e46e06a15c9e162abea2 WatchSource:0}: Error finding container c846d35166eec06c56b3b14746bc8da097f9becbb0a1e46e06a15c9e162abea2: Status 404 returned error can't find the container with id c846d35166eec06c56b3b14746bc8da097f9becbb0a1e46e06a15c9e162abea2 Nov 03 14:54:14.826460 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:14.826389 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" event={"ID":"120469ce-b114-4f5f-b04e-cda3add1ffa2","Type":"ContainerStarted","Data":"c846d35166eec06c56b3b14746bc8da097f9becbb0a1e46e06a15c9e162abea2"} Nov 03 14:54:15.832665 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:15.832627 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" event={"ID":"120469ce-b114-4f5f-b04e-cda3add1ffa2","Type":"ContainerStarted","Data":"c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13"} Nov 03 14:54:15.833172 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:15.832757 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:15.875302 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:15.875249 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" podStartSLOduration=2.510974696 podStartE2EDuration="2.875235484s" totalImagesPullingTime="364.260788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:15.875110102 +0000 UTC m=+2531.433746586" watchObservedRunningTime="2026-11-03 14:54:15.875235484 +0000 UTC m=+2531.433871965" Nov 03 14:54:21.523567 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.523532 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6"] Nov 03 14:54:21.524037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.524021 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd70ba0-113b-4458-afc4-5816a374036c" containerName="istio-proxy" Nov 03 14:54:21.527201 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.527183 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" Nov 03 14:54:21.538993 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.538968 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk8fg\" (UniqueName: \"kubernetes.io/projected/8d50a8e0-b377-40f1-bd99-8bd8079a8755-kube-api-access-jk8fg\") pod \"payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6\" (UID: \"8d50a8e0-b377-40f1-bd99-8bd8079a8755\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" Nov 03 14:54:21.539092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.539015 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d50a8e0-b377-40f1-bd99-8bd8079a8755-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6\" (UID: \"8d50a8e0-b377-40f1-bd99-8bd8079a8755\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" Nov 03 14:54:21.557034 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.557006 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-72b663-dockercfg-nfkm9\"" Nov 03 14:54:21.557183 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.557048 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-72b663\"" Nov 03 14:54:21.558731 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.558711 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6"] Nov 03 14:54:21.640421 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.640386 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8fg\" (UniqueName: \"kubernetes.io/projected/8d50a8e0-b377-40f1-bd99-8bd8079a8755-kube-api-access-jk8fg\") pod \"payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6\" (UID: \"8d50a8e0-b377-40f1-bd99-8bd8079a8755\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" Nov 03 14:54:21.640630 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.640444 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d50a8e0-b377-40f1-bd99-8bd8079a8755-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6\" (UID: \"8d50a8e0-b377-40f1-bd99-8bd8079a8755\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" Nov 03 14:54:21.641165 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.641145 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d50a8e0-b377-40f1-bd99-8bd8079a8755-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6\" (UID: \"8d50a8e0-b377-40f1-bd99-8bd8079a8755\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" Nov 03 14:54:21.665408 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.665371 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk8fg\" (UniqueName: \"kubernetes.io/projected/8d50a8e0-b377-40f1-bd99-8bd8079a8755-kube-api-access-jk8fg\") pod \"payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6\" (UID: \"8d50a8e0-b377-40f1-bd99-8bd8079a8755\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" Nov 03 14:54:21.840977 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.840903 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" Nov 03 14:54:21.842442 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:21.842421 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:54:22.143090 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:22.143068 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6"] Nov 03 14:54:22.145214 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:54:22.145178 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d50a8e0_b377_40f1_bd99_8bd8079a8755.slice/crio-6ab0f56271f66f33d9ccbabc274988da38ae24f2674dca02eb902e88f488721e WatchSource:0}: Error finding container 6ab0f56271f66f33d9ccbabc274988da38ae24f2674dca02eb902e88f488721e: Status 404 returned error can't find the container with id 6ab0f56271f66f33d9ccbabc274988da38ae24f2674dca02eb902e88f488721e Nov 03 14:54:22.882650 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:22.882594 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" event={"ID":"8d50a8e0-b377-40f1-bd99-8bd8079a8755","Type":"ContainerStarted","Data":"6ab0f56271f66f33d9ccbabc274988da38ae24f2674dca02eb902e88f488721e"} Nov 03 14:54:23.889770 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:23.889735 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" event={"ID":"8d50a8e0-b377-40f1-bd99-8bd8079a8755","Type":"ContainerStarted","Data":"0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5"} Nov 03 14:54:23.890151 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:23.889799 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" Nov 03 14:54:23.932924 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:23.932877 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" podStartSLOduration=1.341316549 podStartE2EDuration="2.932863154s" totalImagesPullingTime="1.591546605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:23.932655221 +0000 UTC m=+2539.491291696" watchObservedRunningTime="2026-11-03 14:54:23.932863154 +0000 UTC m=+2539.491499643" Nov 03 14:54:28.373351 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:28.373312 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:54:34.897504 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:34.897477 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" Nov 03 14:54:44.429724 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.429690 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9"] Nov 03 14:54:44.433426 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.433407 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" Nov 03 14:54:44.462056 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.462021 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-131649\"" Nov 03 14:54:44.462056 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.462047 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-131649-dockercfg-wg4hx\"" Nov 03 14:54:44.463860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.463838 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9"] Nov 03 14:54:44.491934 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.491899 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh"] Nov 03 14:54:44.496117 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.496087 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" Nov 03 14:54:44.534664 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.516484 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh"] Nov 03 14:54:44.573487 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.573456 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfvsq\" (UniqueName: \"kubernetes.io/projected/21d742c8-a927-435f-954a-bf19ce280cdb-kube-api-access-dfvsq\") pod \"payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9\" (UID: \"21d742c8-a927-435f-954a-bf19ce280cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" Nov 03 14:54:44.573697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.573566 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9\" (UID: \"21d742c8-a927-435f-954a-bf19ce280cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" Nov 03 14:54:44.573697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.573618 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59xc\" (UniqueName: \"kubernetes.io/projected/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-kube-api-access-f59xc\") pod \"payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh\" (UID: \"f5de8a8f-82e2-489f-b7ec-7eeb75df763e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" Nov 03 14:54:44.573795 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.573710 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh\" (UID: \"f5de8a8f-82e2-489f-b7ec-7eeb75df763e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" Nov 03 14:54:44.674442 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.674410 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9\" (UID: \"21d742c8-a927-435f-954a-bf19ce280cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" Nov 03 14:54:44.674442 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.674447 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59xc\" (UniqueName: \"kubernetes.io/projected/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-kube-api-access-f59xc\") pod \"payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh\" (UID: \"f5de8a8f-82e2-489f-b7ec-7eeb75df763e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" Nov 03 14:54:44.674706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.674524 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh\" (UID: \"f5de8a8f-82e2-489f-b7ec-7eeb75df763e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" Nov 03 14:54:44.674706 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.674586 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfvsq\" (UniqueName: \"kubernetes.io/projected/21d742c8-a927-435f-954a-bf19ce280cdb-kube-api-access-dfvsq\") pod \"payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9\" (UID: \"21d742c8-a927-435f-954a-bf19ce280cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" Nov 03 14:54:44.675163 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.675143 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9\" (UID: \"21d742c8-a927-435f-954a-bf19ce280cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" Nov 03 14:54:44.675254 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.675236 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh\" (UID: \"f5de8a8f-82e2-489f-b7ec-7eeb75df763e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" Nov 03 14:54:44.697340 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.697260 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59xc\" (UniqueName: \"kubernetes.io/projected/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-kube-api-access-f59xc\") pod \"payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh\" (UID: \"f5de8a8f-82e2-489f-b7ec-7eeb75df763e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" Nov 03 14:54:44.697480 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.697440 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfvsq\" (UniqueName: \"kubernetes.io/projected/21d742c8-a927-435f-954a-bf19ce280cdb-kube-api-access-dfvsq\") pod \"payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9\" (UID: \"21d742c8-a927-435f-954a-bf19ce280cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" Nov 03 14:54:44.746920 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.746883 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" Nov 03 14:54:44.845266 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:44.845223 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" Nov 03 14:54:45.046356 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:45.046334 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9"] Nov 03 14:54:45.048359 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:54:45.048332 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d742c8_a927_435f_954a_bf19ce280cdb.slice/crio-eea7447cc97452a3be4aac32ae9760cf8020ebd86fd3ab584c20709f5072aad8 WatchSource:0}: Error finding container eea7447cc97452a3be4aac32ae9760cf8020ebd86fd3ab584c20709f5072aad8: Status 404 returned error can't find the container with id eea7447cc97452a3be4aac32ae9760cf8020ebd86fd3ab584c20709f5072aad8 Nov 03 14:54:45.144554 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:45.144518 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh"] Nov 03 14:54:45.146914 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:54:45.146888 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5de8a8f_82e2_489f_b7ec_7eeb75df763e.slice/crio-97eccf4a655200aab51f5f022945bae44730fe835f2372fdec51d79a38862208 WatchSource:0}: Error finding container 97eccf4a655200aab51f5f022945bae44730fe835f2372fdec51d79a38862208: Status 404 returned error can't find the container with id 97eccf4a655200aab51f5f022945bae44730fe835f2372fdec51d79a38862208 Nov 03 14:54:45.997894 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:45.997854 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" event={"ID":"f5de8a8f-82e2-489f-b7ec-7eeb75df763e","Type":"ContainerStarted","Data":"852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe"} Nov 03 14:54:45.998351 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:45.997915 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" Nov 03 14:54:45.998351 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:45.997933 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" event={"ID":"f5de8a8f-82e2-489f-b7ec-7eeb75df763e","Type":"ContainerStarted","Data":"97eccf4a655200aab51f5f022945bae44730fe835f2372fdec51d79a38862208"} Nov 03 14:54:45.999456 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:45.999430 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" event={"ID":"21d742c8-a927-435f-954a-bf19ce280cdb","Type":"ContainerStarted","Data":"d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0"} Nov 03 14:54:45.999600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:45.999473 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" event={"ID":"21d742c8-a927-435f-954a-bf19ce280cdb","Type":"ContainerStarted","Data":"eea7447cc97452a3be4aac32ae9760cf8020ebd86fd3ab584c20709f5072aad8"} Nov 03 14:54:45.999600 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:45.999545 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" Nov 03 14:54:46.047791 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.047733 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" podStartSLOduration=2.047719643 podStartE2EDuration="2.047719643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:46.04691073 +0000 UTC m=+2561.605547214" watchObservedRunningTime="2026-11-03 14:54:46.047719643 +0000 UTC m=+2561.606356126" Nov 03 14:54:46.092880 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.092850 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742"] Nov 03 14:54:46.096436 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.096415 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.104629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.104552 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" podStartSLOduration=2.104534935 podStartE2EDuration="2.104534935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:46.102869127 +0000 UTC m=+2561.661505611" watchObservedRunningTime="2026-11-03 14:54:46.104534935 +0000 UTC m=+2561.663171421" Nov 03 14:54:46.124792 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.124746 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-131649-serving-cert\"" Nov 03 14:54:46.126982 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.126951 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742"] Nov 03 14:54:46.189599 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.189564 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-metrics-certs\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.189795 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.189618 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzgf7\" (UniqueName: \"kubernetes.io/projected/28cb3b18-a907-432e-85e3-c93893ce2ecf-kube-api-access-bzgf7\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.189795 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.189747 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.189905 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.189808 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28cb3b18-a907-432e-85e3-c93893ce2ecf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.304039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.302477 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-metrics-certs\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.304039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.302549 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzgf7\" (UniqueName: \"kubernetes.io/projected/28cb3b18-a907-432e-85e3-c93893ce2ecf-kube-api-access-bzgf7\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.304039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.302640 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.304039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.302690 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28cb3b18-a907-432e-85e3-c93893ce2ecf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.304039 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.303579 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28cb3b18-a907-432e-85e3-c93893ce2ecf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.304424 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:46.304394 6389 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-serving-cert: secret "maas-api-e2e-ipp-b-131649-serving-cert" not found Nov 03 14:54:46.304499 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:46.304484 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-maas-api-tls podName:28cb3b18-a907-432e-85e3-c93893ce2ecf nodeName:}" failed. No retries permitted until 2026-11-03 14:54:46.804462048 +0000 UTC m=+2562.363098532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-maas-api-tls") pod "maas-api-e2e-ipp-b-131649-7858579c5f-s2742" (UID: "28cb3b18-a907-432e-85e3-c93893ce2ecf") : secret "maas-api-e2e-ipp-b-131649-serving-cert" not found Nov 03 14:54:46.308232 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.308200 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-metrics-certs\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.326439 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.326411 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzgf7\" (UniqueName: \"kubernetes.io/projected/28cb3b18-a907-432e-85e3-c93893ce2ecf-kube-api-access-bzgf7\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.808829 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.808789 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.811407 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.811386 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-131649-7858579c5f-s2742\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:46.849804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.849771 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh"] Nov 03 14:54:46.883782 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:46.883751 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9"] Nov 03 14:54:47.011543 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:47.011510 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:47.112753 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:47.112717 6389 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-131649: configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:47.112952 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:47.112794 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume podName:21d742c8-a927-435f-954a-bf19ce280cdb nodeName:}" failed. No retries permitted until 2026-11-03 14:54:47.61277363 +0000 UTC m=+2563.171410103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" (UID: "21d742c8-a927-435f-954a-bf19ce280cdb") : configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:47.113668 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:47.113374 6389 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-131649: configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:47.113668 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:47.113433 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume podName:f5de8a8f-82e2-489f-b7ec-7eeb75df763e nodeName:}" failed. No retries permitted until 2026-11-03 14:54:47.613416562 +0000 UTC m=+2563.172053030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume") pod "payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" (UID: "f5de8a8f-82e2-489f-b7ec-7eeb75df763e") : configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:47.318632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:47.318585 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742"] Nov 03 14:54:47.321175 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:54:47.321147 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28cb3b18_a907_432e_85e3_c93893ce2ecf.slice/crio-638ad6693d54b2174d0f91d4aefff6167a4e953621239c2adcf258ebed16df70 WatchSource:0}: Error finding container 638ad6693d54b2174d0f91d4aefff6167a4e953621239c2adcf258ebed16df70: Status 404 returned error can't find the container with id 638ad6693d54b2174d0f91d4aefff6167a4e953621239c2adcf258ebed16df70 Nov 03 14:54:47.617623 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:47.617530 6389 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-131649: configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:47.617623 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:47.617602 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume podName:21d742c8-a927-435f-954a-bf19ce280cdb nodeName:}" failed. No retries permitted until 2026-11-03 14:54:48.617589306 +0000 UTC m=+2564.176225769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" (UID: "21d742c8-a927-435f-954a-bf19ce280cdb") : configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:47.617884 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:47.617531 6389 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-131649: configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:47.617884 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:47.617670 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume podName:f5de8a8f-82e2-489f-b7ec-7eeb75df763e nodeName:}" failed. No retries permitted until 2026-11-03 14:54:48.617660171 +0000 UTC m=+2564.176296633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume") pod "payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" (UID: "f5de8a8f-82e2-489f-b7ec-7eeb75df763e") : configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:48.010946 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:48.010899 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" event={"ID":"28cb3b18-a907-432e-85e3-c93893ce2ecf","Type":"ContainerStarted","Data":"f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f"} Nov 03 14:54:48.011165 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:48.011096 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:48.011165 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:48.011128 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" event={"ID":"28cb3b18-a907-432e-85e3-c93893ce2ecf","Type":"ContainerStarted","Data":"638ad6693d54b2174d0f91d4aefff6167a4e953621239c2adcf258ebed16df70"} Nov 03 14:54:48.054931 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:48.054883 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" podStartSLOduration=1.691274114 podStartE2EDuration="2.054869233s" totalImagesPullingTime="363.595119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:48.053660103 +0000 UTC m=+2563.612296589" watchObservedRunningTime="2026-11-03 14:54:48.054869233 +0000 UTC m=+2563.613505747" Nov 03 14:54:48.627025 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:48.626982 6389 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-131649: configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:48.627025 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:48.627020 6389 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-131649: configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:48.627311 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:48.627065 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume podName:f5de8a8f-82e2-489f-b7ec-7eeb75df763e nodeName:}" failed. No retries permitted until 2026-11-03 14:54:50.62704757 +0000 UTC m=+2566.185684045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume") pod "payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" (UID: "f5de8a8f-82e2-489f-b7ec-7eeb75df763e") : configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:48.627311 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:48.627085 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume podName:21d742c8-a927-435f-954a-bf19ce280cdb nodeName:}" failed. No retries permitted until 2026-11-03 14:54:50.627077278 +0000 UTC m=+2566.185713740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" (UID: "21d742c8-a927-435f-954a-bf19ce280cdb") : configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:50.653721 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:50.653690 6389 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-131649: configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:50.654148 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:50.653766 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume podName:21d742c8-a927-435f-954a-bf19ce280cdb nodeName:}" failed. No retries permitted until 2026-11-03 14:54:54.653745591 +0000 UTC m=+2570.212382072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" (UID: "21d742c8-a927-435f-954a-bf19ce280cdb") : configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:50.654148 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:50.653785 6389 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-131649: configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:50.654148 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:50.653828 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume podName:f5de8a8f-82e2-489f-b7ec-7eeb75df763e nodeName:}" failed. No retries permitted until 2026-11-03 14:54:54.653817948 +0000 UTC m=+2570.212454418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume") pod "payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" (UID: "f5de8a8f-82e2-489f-b7ec-7eeb75df763e") : configmap "payload-processing-plugins-e2e-ipp-b-131649" not found Nov 03 14:54:53.011436 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.011402 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" podUID="21d742c8-a927-435f-954a-bf19ce280cdb" containerName="payload-pre-processing" containerID="cri-o://d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0" gracePeriod=25 Nov 03 14:54:53.011436 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.011394 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" podUID="f5de8a8f-82e2-489f-b7ec-7eeb75df763e" containerName="payload-processing" containerID="cri-o://852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe" gracePeriod=25 Nov 03 14:54:53.289262 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.289238 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" Nov 03 14:54:53.292297 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.292276 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" Nov 03 14:54:53.379037 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.379006 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume\") pod \"21d742c8-a927-435f-954a-bf19ce280cdb\" (UID: \"21d742c8-a927-435f-954a-bf19ce280cdb\") " Nov 03 14:54:53.379268 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.379191 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d742c8-a927-435f-954a-bf19ce280cdb-kube-api-access-dfvsq\" (UniqueName: \"kubernetes.io/projected/21d742c8-a927-435f-954a-bf19ce280cdb-kube-api-access-dfvsq\") pod \"21d742c8-a927-435f-954a-bf19ce280cdb\" (UID: \"21d742c8-a927-435f-954a-bf19ce280cdb\") " Nov 03 14:54:53.379757 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.379726 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume" pod "21d742c8-a927-435f-954a-bf19ce280cdb" (UID: "21d742c8-a927-435f-954a-bf19ce280cdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:54:53.381596 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.381572 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d742c8-a927-435f-954a-bf19ce280cdb-kube-api-access-dfvsq" pod "21d742c8-a927-435f-954a-bf19ce280cdb" (UID: "21d742c8-a927-435f-954a-bf19ce280cdb"). InnerVolumeSpecName "kube-api-access-dfvsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:54:53.480078 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.480046 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume\") pod \"f5de8a8f-82e2-489f-b7ec-7eeb75df763e\" (UID: \"f5de8a8f-82e2-489f-b7ec-7eeb75df763e\") " Nov 03 14:54:53.480246 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.480094 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-kube-api-access-f59xc\" (UniqueName: \"kubernetes.io/projected/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-kube-api-access-f59xc\") pod \"f5de8a8f-82e2-489f-b7ec-7eeb75df763e\" (UID: \"f5de8a8f-82e2-489f-b7ec-7eeb75df763e\") " Nov 03 14:54:53.480329 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.480316 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfvsq\" (UniqueName: \"kubernetes.io/projected/21d742c8-a927-435f-954a-bf19ce280cdb-kube-api-access-dfvsq\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:54:53.480368 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.480331 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d742c8-a927-435f-954a-bf19ce280cdb-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:54:53.480518 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.480493 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume" pod "f5de8a8f-82e2-489f-b7ec-7eeb75df763e" (UID: "f5de8a8f-82e2-489f-b7ec-7eeb75df763e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:54:53.482361 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.482342 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-kube-api-access-f59xc" pod "f5de8a8f-82e2-489f-b7ec-7eeb75df763e" (UID: "f5de8a8f-82e2-489f-b7ec-7eeb75df763e"). InnerVolumeSpecName "kube-api-access-f59xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:54:53.581402 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.581324 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:54:53.581402 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:53.581352 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f59xc\" (UniqueName: \"kubernetes.io/projected/f5de8a8f-82e2-489f-b7ec-7eeb75df763e-kube-api-access-f59xc\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:54:54.030740 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.030473 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:54:54.045239 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.045201 6389 generic.go:356] "Generic (PLEG): container finished" podID="21d742c8-a927-435f-954a-bf19ce280cdb" containerID="d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0" exitCode=0 Nov 03 14:54:54.045422 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.045363 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" event={"ID":"21d742c8-a927-435f-954a-bf19ce280cdb","Type":"ContainerDied","Data":"d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0"} Nov 03 14:54:54.045422 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.045413 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" event={"ID":"21d742c8-a927-435f-954a-bf19ce280cdb","Type":"ContainerDied","Data":"eea7447cc97452a3be4aac32ae9760cf8020ebd86fd3ab584c20709f5072aad8"} Nov 03 14:54:54.045536 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.045440 6389 scope.go:122] "RemoveContainer" containerID="d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0" Nov 03 14:54:54.045688 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.045643 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9" Nov 03 14:54:54.058144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.057596 6389 generic.go:356] "Generic (PLEG): container finished" podID="f5de8a8f-82e2-489f-b7ec-7eeb75df763e" containerID="852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe" exitCode=0 Nov 03 14:54:54.058144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.057706 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" event={"ID":"f5de8a8f-82e2-489f-b7ec-7eeb75df763e","Type":"ContainerDied","Data":"852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe"} Nov 03 14:54:54.058144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.057747 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" Nov 03 14:54:54.058144 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.057757 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh" event={"ID":"f5de8a8f-82e2-489f-b7ec-7eeb75df763e","Type":"ContainerDied","Data":"97eccf4a655200aab51f5f022945bae44730fe835f2372fdec51d79a38862208"} Nov 03 14:54:54.070291 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.070265 6389 scope.go:122] "RemoveContainer" containerID="d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0" Nov 03 14:54:54.071867 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:54.071835 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0\": container with ID starting with d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0 not found: ID does not exist" containerID="d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0" Nov 03 14:54:54.071969 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.071903 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0"} err="failed to get container status \"d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0\": rpc error: code = NotFound desc = could not find container \"d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0\": container with ID starting with d69731c63b81decf63504972d28c9ec870e7ec553dfa7699fa6671337ee2aed0 not found: ID does not exist" Nov 03 14:54:54.071969 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.071927 6389 scope.go:122] "RemoveContainer" containerID="852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe" Nov 03 14:54:54.081895 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.081875 6389 scope.go:122] "RemoveContainer" containerID="852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe" Nov 03 14:54:54.082177 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:54.082149 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe\": container with ID starting with 852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe not found: ID does not exist" containerID="852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe" Nov 03 14:54:54.082231 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.082185 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe"} err="failed to get container status \"852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe\": rpc error: code = NotFound desc = could not find container \"852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe\": container with ID starting with 852bb711c5e20699432b5392b7158c7329ac2ee3659fbb4c0badcadb901f2fbe not found: ID does not exist" Nov 03 14:54:54.116439 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.116409 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9"] Nov 03 14:54:54.123272 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.123249 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-5ff5bd9cb7-2gst9"] Nov 03 14:54:54.160972 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.160943 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh"] Nov 03 14:54:54.168961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.168936 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-131649-5c9644669f-9jkkh"] Nov 03 14:54:54.459675 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.459642 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6"] Nov 03 14:54:54.460059 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.460045 6389 cpu_manager.go:414] "containerMap: removing container" podUID="21d742c8-a927-435f-954a-bf19ce280cdb" containerName="payload-pre-processing" Nov 03 14:54:54.460109 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.460060 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d742c8-a927-435f-954a-bf19ce280cdb" containerName="payload-pre-processing" Nov 03 14:54:54.460109 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.460073 6389 cpu_manager.go:414] "containerMap: removing container" podUID="f5de8a8f-82e2-489f-b7ec-7eeb75df763e" containerName="payload-processing" Nov 03 14:54:54.460109 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.460078 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5de8a8f-82e2-489f-b7ec-7eeb75df763e" containerName="payload-processing" Nov 03 14:54:54.460214 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.460164 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d742c8-a927-435f-954a-bf19ce280cdb" containerName="payload-pre-processing" Nov 03 14:54:54.460214 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.460173 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5de8a8f-82e2-489f-b7ec-7eeb75df763e" containerName="payload-processing" Nov 03 14:54:54.464641 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.464600 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" Nov 03 14:54:54.484859 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.484835 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-131649-dockercfg-85jmk\"" Nov 03 14:54:54.485798 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.485774 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6"] Nov 03 14:54:54.555236 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.555195 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-131649\"" Nov 03 14:54:54.592223 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.592192 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zwq\" (UniqueName: \"kubernetes.io/projected/2da2ba2e-e4da-4265-bae7-674911c08856-kube-api-access-x8zwq\") pod \"payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6\" (UID: \"2da2ba2e-e4da-4265-bae7-674911c08856\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" Nov 03 14:54:54.592382 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.592331 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da2ba2e-e4da-4265-bae7-674911c08856-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6\" (UID: \"2da2ba2e-e4da-4265-bae7-674911c08856\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" Nov 03 14:54:54.617149 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.617119 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d742c8-a927-435f-954a-bf19ce280cdb" path="/var/lib/kubelet/pods/21d742c8-a927-435f-954a-bf19ce280cdb/volumes" Nov 03 14:54:54.617481 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.617468 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5de8a8f-82e2-489f-b7ec-7eeb75df763e" path="/var/lib/kubelet/pods/f5de8a8f-82e2-489f-b7ec-7eeb75df763e/volumes" Nov 03 14:54:54.693818 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.693790 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zwq\" (UniqueName: \"kubernetes.io/projected/2da2ba2e-e4da-4265-bae7-674911c08856-kube-api-access-x8zwq\") pod \"payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6\" (UID: \"2da2ba2e-e4da-4265-bae7-674911c08856\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" Nov 03 14:54:54.693990 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.693882 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da2ba2e-e4da-4265-bae7-674911c08856-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6\" (UID: \"2da2ba2e-e4da-4265-bae7-674911c08856\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" Nov 03 14:54:54.695398 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.695358 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da2ba2e-e4da-4265-bae7-674911c08856-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6\" (UID: \"2da2ba2e-e4da-4265-bae7-674911c08856\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" Nov 03 14:54:54.716234 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.716168 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zwq\" (UniqueName: \"kubernetes.io/projected/2da2ba2e-e4da-4265-bae7-674911c08856-kube-api-access-x8zwq\") pod \"payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6\" (UID: \"2da2ba2e-e4da-4265-bae7-674911c08856\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" Nov 03 14:54:54.778206 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:54.778174 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" Nov 03 14:54:55.078565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:55.078460 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6"] Nov 03 14:54:55.081307 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:54:55.081272 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da2ba2e_e4da_4265_bae7_674911c08856.slice/crio-2b3f33dd18298c63fcd49e3bc1c83384933d8c720f06c23b5258ec5e01b63c2d WatchSource:0}: Error finding container 2b3f33dd18298c63fcd49e3bc1c83384933d8c720f06c23b5258ec5e01b63c2d: Status 404 returned error can't find the container with id 2b3f33dd18298c63fcd49e3bc1c83384933d8c720f06c23b5258ec5e01b63c2d Nov 03 14:54:56.071466 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:56.071422 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" event={"ID":"2da2ba2e-e4da-4265-bae7-674911c08856","Type":"ContainerStarted","Data":"cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9"} Nov 03 14:54:56.071660 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:56.071514 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" Nov 03 14:54:56.071660 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:56.071540 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" event={"ID":"2da2ba2e-e4da-4265-bae7-674911c08856","Type":"ContainerStarted","Data":"2b3f33dd18298c63fcd49e3bc1c83384933d8c720f06c23b5258ec5e01b63c2d"} Nov 03 14:54:56.116517 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:54:56.116462 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" podStartSLOduration=2.1164485 podStartE2EDuration="2.1164485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:56.115689333 +0000 UTC m=+2571.674325816" watchObservedRunningTime="2026-11-03 14:54:56.1164485 +0000 UTC m=+2571.675084984" Nov 03 14:54:58.372892 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:54:58.372861 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:55:07.079102 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:55:07.079069 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" Nov 03 14:55:14.584176 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:55:14.584142 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r9bfq_a41e843c-6800-4404-a7cb-adb5af6493f9/payload-processing/0.log" Nov 03 14:55:14.801000 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:55:14.800972 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r9bfq_a41e843c-6800-4404-a7cb-adb5af6493f9/payload-processing/0.log" Nov 03 14:55:15.018916 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:55:15.018873 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6_2da2ba2e-e4da-4265-bae7-674911c08856/payload-processing/0.log" Nov 03 14:55:28.373920 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:55:28.373882 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:55:54.062119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:55:54.062037 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r9bfq_a41e843c-6800-4404-a7cb-adb5af6493f9/payload-processing/0.log" Nov 03 14:55:54.274183 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:55:54.274154 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r9bfq_a41e843c-6800-4404-a7cb-adb5af6493f9/payload-processing/0.log" Nov 03 14:55:54.472190 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:55:54.472125 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6_2da2ba2e-e4da-4265-bae7-674911c08856/payload-processing/0.log" Nov 03 14:55:58.373387 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:55:58.373351 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:56:28.373277 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:56:28.373230 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:56:47.513161 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:47.513118 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6"] Nov 03 14:56:47.513677 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:47.513407 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" podUID="120469ce-b114-4f5f-b04e-cda3add1ffa2" containerName="maas-api" containerID="cri-o://c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13" gracePeriod=30 Nov 03 14:56:47.731989 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:47.731957 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6"] Nov 03 14:56:47.732230 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:47.732206 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" podUID="8d50a8e0-b377-40f1-bd99-8bd8079a8755" containerName="payload-pre-processing" containerID="cri-o://0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5" gracePeriod=30 Nov 03 14:56:47.998565 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:47.998534 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" Nov 03 14:56:48.025745 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.025714 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:56:48.159668 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.159566 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d50a8e0-b377-40f1-bd99-8bd8079a8755-kube-api-access-jk8fg\" (UniqueName: \"kubernetes.io/projected/8d50a8e0-b377-40f1-bd99-8bd8079a8755-kube-api-access-jk8fg\") pod \"8d50a8e0-b377-40f1-bd99-8bd8079a8755\" (UID: \"8d50a8e0-b377-40f1-bd99-8bd8079a8755\") " Nov 03 14:56:48.159668 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.159663 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-metrics-certs\") pod \"120469ce-b114-4f5f-b04e-cda3add1ffa2\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " Nov 03 14:56:48.159898 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.159699 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-maas-api-tls\") pod \"120469ce-b114-4f5f-b04e-cda3add1ffa2\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " Nov 03 14:56:48.159898 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.159719 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d50a8e0-b377-40f1-bd99-8bd8079a8755-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d50a8e0-b377-40f1-bd99-8bd8079a8755-plugins-config-volume\") pod \"8d50a8e0-b377-40f1-bd99-8bd8079a8755\" (UID: \"8d50a8e0-b377-40f1-bd99-8bd8079a8755\") " Nov 03 14:56:48.159898 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.159784 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/120469ce-b114-4f5f-b04e-cda3add1ffa2-kube-api-access-rs497\" (UniqueName: \"kubernetes.io/projected/120469ce-b114-4f5f-b04e-cda3add1ffa2-kube-api-access-rs497\") pod \"120469ce-b114-4f5f-b04e-cda3add1ffa2\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " Nov 03 14:56:48.159898 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.159825 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/120469ce-b114-4f5f-b04e-cda3add1ffa2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/120469ce-b114-4f5f-b04e-cda3add1ffa2-openshift-service-ca\") pod \"120469ce-b114-4f5f-b04e-cda3add1ffa2\" (UID: \"120469ce-b114-4f5f-b04e-cda3add1ffa2\") " Nov 03 14:56:48.160266 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.160243 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/120469ce-b114-4f5f-b04e-cda3add1ffa2-openshift-service-ca" pod "120469ce-b114-4f5f-b04e-cda3add1ffa2" (UID: "120469ce-b114-4f5f-b04e-cda3add1ffa2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:56:48.160357 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.160254 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d50a8e0-b377-40f1-bd99-8bd8079a8755-plugins-config-volume" pod "8d50a8e0-b377-40f1-bd99-8bd8079a8755" (UID: "8d50a8e0-b377-40f1-bd99-8bd8079a8755"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:56:48.162163 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.162133 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/120469ce-b114-4f5f-b04e-cda3add1ffa2-kube-api-access-rs497" pod "120469ce-b114-4f5f-b04e-cda3add1ffa2" (UID: "120469ce-b114-4f5f-b04e-cda3add1ffa2"). InnerVolumeSpecName "kube-api-access-rs497". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:56:48.162262 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.162176 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-maas-api-tls" pod "120469ce-b114-4f5f-b04e-cda3add1ffa2" (UID: "120469ce-b114-4f5f-b04e-cda3add1ffa2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:56:48.162440 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.162423 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-metrics-certs" pod "120469ce-b114-4f5f-b04e-cda3add1ffa2" (UID: "120469ce-b114-4f5f-b04e-cda3add1ffa2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:56:48.162485 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.162445 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d50a8e0-b377-40f1-bd99-8bd8079a8755-kube-api-access-jk8fg" pod "8d50a8e0-b377-40f1-bd99-8bd8079a8755" (UID: "8d50a8e0-b377-40f1-bd99-8bd8079a8755"). InnerVolumeSpecName "kube-api-access-jk8fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:56:48.261018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.260982 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:56:48.261018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.261011 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d50a8e0-b377-40f1-bd99-8bd8079a8755-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:56:48.261018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.261021 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs497\" (UniqueName: \"kubernetes.io/projected/120469ce-b114-4f5f-b04e-cda3add1ffa2-kube-api-access-rs497\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:56:48.261018 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.261030 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/120469ce-b114-4f5f-b04e-cda3add1ffa2-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:56:48.261304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.261038 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk8fg\" (UniqueName: \"kubernetes.io/projected/8d50a8e0-b377-40f1-bd99-8bd8079a8755-kube-api-access-jk8fg\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:56:48.261304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.261047 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/120469ce-b114-4f5f-b04e-cda3add1ffa2-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:56:48.620301 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.620260 6389 generic.go:356] "Generic (PLEG): container finished" podID="120469ce-b114-4f5f-b04e-cda3add1ffa2" containerID="c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13" exitCode=0 Nov 03 14:56:48.620301 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.620304 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" Nov 03 14:56:48.620890 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.620346 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" event={"ID":"120469ce-b114-4f5f-b04e-cda3add1ffa2","Type":"ContainerDied","Data":"c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13"} Nov 03 14:56:48.620890 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.620396 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6" event={"ID":"120469ce-b114-4f5f-b04e-cda3add1ffa2","Type":"ContainerDied","Data":"c846d35166eec06c56b3b14746bc8da097f9becbb0a1e46e06a15c9e162abea2"} Nov 03 14:56:48.620890 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.620415 6389 scope.go:122] "RemoveContainer" containerID="c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13" Nov 03 14:56:48.622356 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.622303 6389 generic.go:356] "Generic (PLEG): container finished" podID="8d50a8e0-b377-40f1-bd99-8bd8079a8755" containerID="0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5" exitCode=0 Nov 03 14:56:48.622437 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.622359 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" event={"ID":"8d50a8e0-b377-40f1-bd99-8bd8079a8755","Type":"ContainerDied","Data":"0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5"} Nov 03 14:56:48.622437 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.622395 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" event={"ID":"8d50a8e0-b377-40f1-bd99-8bd8079a8755","Type":"ContainerDied","Data":"6ab0f56271f66f33d9ccbabc274988da38ae24f2674dca02eb902e88f488721e"} Nov 03 14:56:48.622550 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.622461 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6" Nov 03 14:56:48.632317 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.632297 6389 scope.go:122] "RemoveContainer" containerID="c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13" Nov 03 14:56:48.632896 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:56:48.632872 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13\": container with ID starting with c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13 not found: ID does not exist" containerID="c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13" Nov 03 14:56:48.633005 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.632907 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13"} err="failed to get container status \"c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13\": rpc error: code = NotFound desc = could not find container \"c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13\": container with ID starting with c132997be8785453c2d106778499d1dc50cb6ee863438d6c7f12b47987a32f13 not found: ID does not exist" Nov 03 14:56:48.633005 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.632924 6389 scope.go:122] "RemoveContainer" containerID="0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5" Nov 03 14:56:48.641431 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.641415 6389 scope.go:122] "RemoveContainer" containerID="0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5" Nov 03 14:56:48.641706 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:56:48.641685 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5\": container with ID starting with 0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5 not found: ID does not exist" containerID="0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5" Nov 03 14:56:48.641785 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.641713 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5"} err="failed to get container status \"0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5\": rpc error: code = NotFound desc = could not find container \"0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5\": container with ID starting with 0835b89307800f152d126024786d58fe04e0586a2e09cb66e71821f67d5656f5 not found: ID does not exist" Nov 03 14:56:48.698583 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.698546 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6"] Nov 03 14:56:48.713984 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.713951 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-72b663-575c54bf74-fdch6"] Nov 03 14:56:48.756659 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.756632 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6"] Nov 03 14:56:48.765218 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:48.765197 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-7698d78955-r4gg6"] Nov 03 14:56:50.616842 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:50.616813 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="120469ce-b114-4f5f-b04e-cda3add1ffa2" path="/var/lib/kubelet/pods/120469ce-b114-4f5f-b04e-cda3add1ffa2/volumes" Nov 03 14:56:50.617242 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:56:50.617178 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d50a8e0-b377-40f1-bd99-8bd8079a8755" path="/var/lib/kubelet/pods/8d50a8e0-b377-40f1-bd99-8bd8079a8755/volumes" Nov 03 14:56:58.373859 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:56:58.373818 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:57:10.653322 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:10.653294 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742"] Nov 03 14:57:10.654455 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:10.654421 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" podUID="28cb3b18-a907-432e-85e3-c93893ce2ecf" containerName="maas-api" containerID="cri-o://f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f" gracePeriod=30 Nov 03 14:57:10.841914 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:10.841293 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6"] Nov 03 14:57:10.841914 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:10.841551 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" podUID="2da2ba2e-e4da-4265-bae7-674911c08856" containerName="payload-processing" containerID="cri-o://cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9" gracePeriod=30 Nov 03 14:57:10.967470 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:10.967445 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:57:11.092919 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.092892 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28cb3b18-a907-432e-85e3-c93893ce2ecf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28cb3b18-a907-432e-85e3-c93893ce2ecf-openshift-service-ca\") pod \"28cb3b18-a907-432e-85e3-c93893ce2ecf\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " Nov 03 14:57:11.093056 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.093010 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-metrics-certs\") pod \"28cb3b18-a907-432e-85e3-c93893ce2ecf\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " Nov 03 14:57:11.093110 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.093098 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28cb3b18-a907-432e-85e3-c93893ce2ecf-kube-api-access-bzgf7\" (UniqueName: \"kubernetes.io/projected/28cb3b18-a907-432e-85e3-c93893ce2ecf-kube-api-access-bzgf7\") pod \"28cb3b18-a907-432e-85e3-c93893ce2ecf\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " Nov 03 14:57:11.093157 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.093141 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-maas-api-tls\") pod \"28cb3b18-a907-432e-85e3-c93893ce2ecf\" (UID: \"28cb3b18-a907-432e-85e3-c93893ce2ecf\") " Nov 03 14:57:11.093285 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.093262 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28cb3b18-a907-432e-85e3-c93893ce2ecf-openshift-service-ca" pod "28cb3b18-a907-432e-85e3-c93893ce2ecf" (UID: "28cb3b18-a907-432e-85e3-c93893ce2ecf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:57:11.093504 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.093485 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28cb3b18-a907-432e-85e3-c93893ce2ecf-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:57:11.095235 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.095214 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28cb3b18-a907-432e-85e3-c93893ce2ecf-kube-api-access-bzgf7" pod "28cb3b18-a907-432e-85e3-c93893ce2ecf" (UID: "28cb3b18-a907-432e-85e3-c93893ce2ecf"). InnerVolumeSpecName "kube-api-access-bzgf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:57:11.095332 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.095240 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-metrics-certs" pod "28cb3b18-a907-432e-85e3-c93893ce2ecf" (UID: "28cb3b18-a907-432e-85e3-c93893ce2ecf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:57:11.095332 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.095270 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-maas-api-tls" pod "28cb3b18-a907-432e-85e3-c93893ce2ecf" (UID: "28cb3b18-a907-432e-85e3-c93893ce2ecf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:57:11.096266 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.096250 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" Nov 03 14:57:11.194801 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.194719 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da2ba2e-e4da-4265-bae7-674911c08856-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da2ba2e-e4da-4265-bae7-674911c08856-plugins-config-volume\") pod \"2da2ba2e-e4da-4265-bae7-674911c08856\" (UID: \"2da2ba2e-e4da-4265-bae7-674911c08856\") " Nov 03 14:57:11.194801 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.194756 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da2ba2e-e4da-4265-bae7-674911c08856-kube-api-access-x8zwq\" (UniqueName: \"kubernetes.io/projected/2da2ba2e-e4da-4265-bae7-674911c08856-kube-api-access-x8zwq\") pod \"2da2ba2e-e4da-4265-bae7-674911c08856\" (UID: \"2da2ba2e-e4da-4265-bae7-674911c08856\") " Nov 03 14:57:11.195049 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.194954 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzgf7\" (UniqueName: \"kubernetes.io/projected/28cb3b18-a907-432e-85e3-c93893ce2ecf-kube-api-access-bzgf7\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:57:11.195049 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.194968 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:57:11.195049 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.194976 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28cb3b18-a907-432e-85e3-c93893ce2ecf-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:57:11.195241 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.195080 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da2ba2e-e4da-4265-bae7-674911c08856-plugins-config-volume" pod "2da2ba2e-e4da-4265-bae7-674911c08856" (UID: "2da2ba2e-e4da-4265-bae7-674911c08856"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:57:11.197127 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.197094 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da2ba2e-e4da-4265-bae7-674911c08856-kube-api-access-x8zwq" pod "2da2ba2e-e4da-4265-bae7-674911c08856" (UID: "2da2ba2e-e4da-4265-bae7-674911c08856"). InnerVolumeSpecName "kube-api-access-x8zwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:57:11.296340 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.296308 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da2ba2e-e4da-4265-bae7-674911c08856-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:57:11.296340 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.296335 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8zwq\" (UniqueName: \"kubernetes.io/projected/2da2ba2e-e4da-4265-bae7-674911c08856-kube-api-access-x8zwq\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:57:11.756444 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.756417 6389 generic.go:356] "Generic (PLEG): container finished" podID="2da2ba2e-e4da-4265-bae7-674911c08856" containerID="cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9" exitCode=0 Nov 03 14:57:11.756935 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.756464 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" Nov 03 14:57:11.756935 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.756503 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" event={"ID":"2da2ba2e-e4da-4265-bae7-674911c08856","Type":"ContainerDied","Data":"cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9"} Nov 03 14:57:11.756935 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.756548 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6" event={"ID":"2da2ba2e-e4da-4265-bae7-674911c08856","Type":"ContainerDied","Data":"2b3f33dd18298c63fcd49e3bc1c83384933d8c720f06c23b5258ec5e01b63c2d"} Nov 03 14:57:11.756935 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.756575 6389 scope.go:122] "RemoveContainer" containerID="cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9" Nov 03 14:57:11.758108 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.758087 6389 generic.go:356] "Generic (PLEG): container finished" podID="28cb3b18-a907-432e-85e3-c93893ce2ecf" containerID="f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f" exitCode=0 Nov 03 14:57:11.758217 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.758136 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" event={"ID":"28cb3b18-a907-432e-85e3-c93893ce2ecf","Type":"ContainerDied","Data":"f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f"} Nov 03 14:57:11.758217 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.758145 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" Nov 03 14:57:11.758217 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.758177 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742" event={"ID":"28cb3b18-a907-432e-85e3-c93893ce2ecf","Type":"ContainerDied","Data":"638ad6693d54b2174d0f91d4aefff6167a4e953621239c2adcf258ebed16df70"} Nov 03 14:57:11.765629 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.765589 6389 scope.go:122] "RemoveContainer" containerID="cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9" Nov 03 14:57:11.765905 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:57:11.765886 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9\": container with ID starting with cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9 not found: ID does not exist" containerID="cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9" Nov 03 14:57:11.765961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.765912 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9"} err="failed to get container status \"cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9\": rpc error: code = NotFound desc = could not find container \"cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9\": container with ID starting with cab6b39083080dd38cdcf1e7bc5385c9a9c3ed3a7dee2f52edbbccd0e16ff7a9 not found: ID does not exist" Nov 03 14:57:11.765961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.765928 6389 scope.go:122] "RemoveContainer" containerID="f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f" Nov 03 14:57:11.774304 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.774289 6389 scope.go:122] "RemoveContainer" containerID="f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f" Nov 03 14:57:11.774580 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:57:11.774564 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f\": container with ID starting with f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f not found: ID does not exist" containerID="f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f" Nov 03 14:57:11.774683 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.774588 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f"} err="failed to get container status \"f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f\": rpc error: code = NotFound desc = could not find container \"f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f\": container with ID starting with f00f4f8ef7b77e1873fdcefe8be10ad46325c3eb181d424635d62d7d5a3b2d0f not found: ID does not exist" Nov 03 14:57:11.816078 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.816049 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6"] Nov 03 14:57:11.821020 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.821000 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-131649-6f6c9596d4-wrjj6"] Nov 03 14:57:11.862253 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.862219 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742"] Nov 03 14:57:11.871490 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:11.871462 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-131649-7858579c5f-s2742"] Nov 03 14:57:12.618664 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:12.618631 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28cb3b18-a907-432e-85e3-c93893ce2ecf" path="/var/lib/kubelet/pods/28cb3b18-a907-432e-85e3-c93893ce2ecf/volumes" Nov 03 14:57:12.619130 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:12.619113 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da2ba2e-e4da-4265-bae7-674911c08856" path="/var/lib/kubelet/pods/2da2ba2e-e4da-4265-bae7-674911c08856/volumes" Nov 03 14:57:19.353356 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353325 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-559f69b56-h5sc2"] Nov 03 14:57:19.353835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353721 6389 cpu_manager.go:414] "containerMap: removing container" podUID="8d50a8e0-b377-40f1-bd99-8bd8079a8755" containerName="payload-pre-processing" Nov 03 14:57:19.353835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353734 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d50a8e0-b377-40f1-bd99-8bd8079a8755" containerName="payload-pre-processing" Nov 03 14:57:19.353835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353749 6389 cpu_manager.go:414] "containerMap: removing container" podUID="2da2ba2e-e4da-4265-bae7-674911c08856" containerName="payload-processing" Nov 03 14:57:19.353835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353755 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da2ba2e-e4da-4265-bae7-674911c08856" containerName="payload-processing" Nov 03 14:57:19.353835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353765 6389 cpu_manager.go:414] "containerMap: removing container" podUID="120469ce-b114-4f5f-b04e-cda3add1ffa2" containerName="maas-api" Nov 03 14:57:19.353835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353770 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="120469ce-b114-4f5f-b04e-cda3add1ffa2" containerName="maas-api" Nov 03 14:57:19.353835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353782 6389 cpu_manager.go:414] "containerMap: removing container" podUID="28cb3b18-a907-432e-85e3-c93893ce2ecf" containerName="maas-api" Nov 03 14:57:19.353835 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353786 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28cb3b18-a907-432e-85e3-c93893ce2ecf" containerName="maas-api" Nov 03 14:57:19.354119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353869 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d50a8e0-b377-40f1-bd99-8bd8079a8755" containerName="payload-pre-processing" Nov 03 14:57:19.354119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353879 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da2ba2e-e4da-4265-bae7-674911c08856" containerName="payload-processing" Nov 03 14:57:19.354119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353885 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="120469ce-b114-4f5f-b04e-cda3add1ffa2" containerName="maas-api" Nov 03 14:57:19.354119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.353893 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="28cb3b18-a907-432e-85e3-c93893ce2ecf" containerName="maas-api" Nov 03 14:57:19.358439 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.358419 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" Nov 03 14:57:19.377109 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.377079 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-559f69b56-h5sc2"] Nov 03 14:57:19.488595 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.488552 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8nrw\" (UniqueName: \"kubernetes.io/projected/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-kube-api-access-x8nrw\") pod \"payload-pre-processing-559f69b56-h5sc2\" (UID: \"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28\") " pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" Nov 03 14:57:19.488804 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.488716 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-plugins-config-volume\") pod \"payload-pre-processing-559f69b56-h5sc2\" (UID: \"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28\") " pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" Nov 03 14:57:19.589374 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.589322 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8nrw\" (UniqueName: \"kubernetes.io/projected/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-kube-api-access-x8nrw\") pod \"payload-pre-processing-559f69b56-h5sc2\" (UID: \"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28\") " pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" Nov 03 14:57:19.589579 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.589411 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-plugins-config-volume\") pod \"payload-pre-processing-559f69b56-h5sc2\" (UID: \"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28\") " pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" Nov 03 14:57:19.590092 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.590072 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-plugins-config-volume\") pod \"payload-pre-processing-559f69b56-h5sc2\" (UID: \"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28\") " pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" Nov 03 14:57:19.612444 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.612377 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8nrw\" (UniqueName: \"kubernetes.io/projected/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-kube-api-access-x8nrw\") pod \"payload-pre-processing-559f69b56-h5sc2\" (UID: \"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28\") " pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" Nov 03 14:57:19.670639 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.670590 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" Nov 03 14:57:19.971912 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:19.971890 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-559f69b56-h5sc2"] Nov 03 14:57:19.974417 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:57:19.974389 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca384f4_cf85_43ee_aa11_0ba7f2ea9d28.slice/crio-6040344cc4926d0828d22e92b05905396d1fd1e98c0d544a34ff505da1fd3bbd WatchSource:0}: Error finding container 6040344cc4926d0828d22e92b05905396d1fd1e98c0d544a34ff505da1fd3bbd: Status 404 returned error can't find the container with id 6040344cc4926d0828d22e92b05905396d1fd1e98c0d544a34ff505da1fd3bbd Nov 03 14:57:20.808498 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:20.808462 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" event={"ID":"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28","Type":"ContainerStarted","Data":"4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e"} Nov 03 14:57:20.808956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:20.808522 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" event={"ID":"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28","Type":"ContainerStarted","Data":"6040344cc4926d0828d22e92b05905396d1fd1e98c0d544a34ff505da1fd3bbd"} Nov 03 14:57:20.808956 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:20.808642 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" Nov 03 14:57:20.852808 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:20.852754 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" podStartSLOduration=1.85273986 podStartE2EDuration="1.85273986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:57:20.851806221 +0000 UTC m=+2716.410442705" watchObservedRunningTime="2026-11-03 14:57:20.85273986 +0000 UTC m=+2716.411376352" Nov 03 14:57:28.373983 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:57:28.373945 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:57:31.816283 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:31.816253 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" Nov 03 14:57:31.891742 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:31.891711 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn"] Nov 03 14:57:36.892386 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:36.892321 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" podUID="a5f57c9a-972c-49a8-a1d8-b703dd4d02a4" containerName="payload-pre-processing" containerID="cri-o://ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36" gracePeriod=25 Nov 03 14:57:37.151424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.151367 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" Nov 03 14:57:37.256522 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.256490 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-kube-api-access-jt88n\" (UniqueName: \"kubernetes.io/projected/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-kube-api-access-jt88n\") pod \"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4\" (UID: \"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4\") " Nov 03 14:57:37.256750 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.256543 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-plugins-config-volume\") pod \"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4\" (UID: \"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4\") " Nov 03 14:57:37.257001 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.256979 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-plugins-config-volume" pod "a5f57c9a-972c-49a8-a1d8-b703dd4d02a4" (UID: "a5f57c9a-972c-49a8-a1d8-b703dd4d02a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:57:37.258925 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.258903 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-kube-api-access-jt88n" pod "a5f57c9a-972c-49a8-a1d8-b703dd4d02a4" (UID: "a5f57c9a-972c-49a8-a1d8-b703dd4d02a4"). InnerVolumeSpecName "kube-api-access-jt88n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:57:37.358177 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.358147 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt88n\" (UniqueName: \"kubernetes.io/projected/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-kube-api-access-jt88n\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:57:37.358177 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.358175 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:57:37.889071 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.889040 6389 generic.go:356] "Generic (PLEG): container finished" podID="a5f57c9a-972c-49a8-a1d8-b703dd4d02a4" containerID="ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36" exitCode=0 Nov 03 14:57:37.889257 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.889085 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" Nov 03 14:57:37.889257 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.889118 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" event={"ID":"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4","Type":"ContainerDied","Data":"ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36"} Nov 03 14:57:37.889257 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.889160 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn" event={"ID":"a5f57c9a-972c-49a8-a1d8-b703dd4d02a4","Type":"ContainerDied","Data":"5bfcf7dd370b0436639600709f6c149e364f25215be0f826ddccead143c7d521"} Nov 03 14:57:37.889257 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.889185 6389 scope.go:122] "RemoveContainer" containerID="ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36" Nov 03 14:57:37.898858 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.898777 6389 scope.go:122] "RemoveContainer" containerID="ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36" Nov 03 14:57:37.899131 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:57:37.899008 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36\": container with ID starting with ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36 not found: ID does not exist" containerID="ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36" Nov 03 14:57:37.899131 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.899049 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36"} err="failed to get container status \"ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36\": rpc error: code = NotFound desc = could not find container \"ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36\": container with ID starting with ab71ecfdc2e2751c5f9dcd5b456858f4afc28d0f50ff5c00504964d4afe59c36 not found: ID does not exist" Nov 03 14:57:37.947029 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.947000 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn"] Nov 03 14:57:37.958932 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:37.958905 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-b9jkn"] Nov 03 14:57:38.617292 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:38.617258 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f57c9a-972c-49a8-a1d8-b703dd4d02a4" path="/var/lib/kubelet/pods/a5f57c9a-972c-49a8-a1d8-b703dd4d02a4/volumes" Nov 03 14:57:46.055414 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:46.055381 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r9bfq"] Nov 03 14:57:48.181351 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.181314 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j"] Nov 03 14:57:48.181806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.181733 6389 cpu_manager.go:414] "containerMap: removing container" podUID="a5f57c9a-972c-49a8-a1d8-b703dd4d02a4" containerName="payload-pre-processing" Nov 03 14:57:48.181806 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.181744 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f57c9a-972c-49a8-a1d8-b703dd4d02a4" containerName="payload-pre-processing" Nov 03 14:57:48.181891 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.181823 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f57c9a-972c-49a8-a1d8-b703dd4d02a4" containerName="payload-pre-processing" Nov 03 14:57:48.186358 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.186336 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" Nov 03 14:57:48.204928 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.204904 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j"] Nov 03 14:57:48.268055 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.268017 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5jzk\" (UniqueName: \"kubernetes.io/projected/8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69-kube-api-access-r5jzk\") pod \"payload-pre-processing-76bfcf8f7d-tzw9j\" (UID: \"8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69\") " pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" Nov 03 14:57:48.268055 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.268061 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69-plugins-config-volume\") pod \"payload-pre-processing-76bfcf8f7d-tzw9j\" (UID: \"8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69\") " pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" Nov 03 14:57:48.369380 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.369344 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jzk\" (UniqueName: \"kubernetes.io/projected/8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69-kube-api-access-r5jzk\") pod \"payload-pre-processing-76bfcf8f7d-tzw9j\" (UID: \"8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69\") " pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" Nov 03 14:57:48.369380 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.369387 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69-plugins-config-volume\") pod \"payload-pre-processing-76bfcf8f7d-tzw9j\" (UID: \"8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69\") " pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" Nov 03 14:57:48.370082 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.370062 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69-plugins-config-volume\") pod \"payload-pre-processing-76bfcf8f7d-tzw9j\" (UID: \"8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69\") " pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" Nov 03 14:57:48.392122 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.392095 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5jzk\" (UniqueName: \"kubernetes.io/projected/8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69-kube-api-access-r5jzk\") pod \"payload-pre-processing-76bfcf8f7d-tzw9j\" (UID: \"8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69\") " pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" Nov 03 14:57:48.498480 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.498389 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" Nov 03 14:57:48.800632 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.800534 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j"] Nov 03 14:57:48.802946 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:57:48.802909 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e00cb7a_21c7_48a1_9c99_ca8a2e3c7c69.slice/crio-87b678428070c3224965f382e16db75da910d0d4fdb3291e4fc0431a3ad279bb WatchSource:0}: Error finding container 87b678428070c3224965f382e16db75da910d0d4fdb3291e4fc0431a3ad279bb: Status 404 returned error can't find the container with id 87b678428070c3224965f382e16db75da910d0d4fdb3291e4fc0431a3ad279bb Nov 03 14:57:48.937930 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.937891 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" event={"ID":"8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69","Type":"ContainerStarted","Data":"bb09ce520f480341e1308b707b006fa2cc86b8b2c0984decd10c57806d52e0ab"} Nov 03 14:57:48.938087 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.937937 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" event={"ID":"8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69","Type":"ContainerStarted","Data":"87b678428070c3224965f382e16db75da910d0d4fdb3291e4fc0431a3ad279bb"} Nov 03 14:57:48.938087 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.937968 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" Nov 03 14:57:48.981620 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:48.981566 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" podStartSLOduration=0.981552121 podStartE2EDuration="981.552121ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:57:48.980835956 +0000 UTC m=+2744.539472440" watchObservedRunningTime="2026-11-03 14:57:48.981552121 +0000 UTC m=+2744.540188604" Nov 03 14:57:51.056010 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.055932 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" podUID="a41e843c-6800-4404-a7cb-adb5af6493f9" containerName="payload-processing" containerID="cri-o://4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c" gracePeriod=25 Nov 03 14:57:51.311870 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.311806 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" Nov 03 14:57:51.395119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.395079 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a41e843c-6800-4404-a7cb-adb5af6493f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41e843c-6800-4404-a7cb-adb5af6493f9-plugins-config-volume\") pod \"a41e843c-6800-4404-a7cb-adb5af6493f9\" (UID: \"a41e843c-6800-4404-a7cb-adb5af6493f9\") " Nov 03 14:57:51.395303 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.395184 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41e843c-6800-4404-a7cb-adb5af6493f9-kube-api-access-6sfzm\" (UniqueName: \"kubernetes.io/projected/a41e843c-6800-4404-a7cb-adb5af6493f9-kube-api-access-6sfzm\") pod \"a41e843c-6800-4404-a7cb-adb5af6493f9\" (UID: \"a41e843c-6800-4404-a7cb-adb5af6493f9\") " Nov 03 14:57:51.395450 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.395429 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41e843c-6800-4404-a7cb-adb5af6493f9-plugins-config-volume" pod "a41e843c-6800-4404-a7cb-adb5af6493f9" (UID: "a41e843c-6800-4404-a7cb-adb5af6493f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:57:51.395540 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.395527 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41e843c-6800-4404-a7cb-adb5af6493f9-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:57:51.397348 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.397322 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41e843c-6800-4404-a7cb-adb5af6493f9-kube-api-access-6sfzm" pod "a41e843c-6800-4404-a7cb-adb5af6493f9" (UID: "a41e843c-6800-4404-a7cb-adb5af6493f9"). InnerVolumeSpecName "kube-api-access-6sfzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:57:51.496639 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.496575 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sfzm\" (UniqueName: \"kubernetes.io/projected/a41e843c-6800-4404-a7cb-adb5af6493f9-kube-api-access-6sfzm\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:57:51.954222 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.954188 6389 generic.go:356] "Generic (PLEG): container finished" podID="a41e843c-6800-4404-a7cb-adb5af6493f9" containerID="4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c" exitCode=0 Nov 03 14:57:51.954425 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.954235 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" Nov 03 14:57:51.954425 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.954275 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" event={"ID":"a41e843c-6800-4404-a7cb-adb5af6493f9","Type":"ContainerDied","Data":"4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c"} Nov 03 14:57:51.954425 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.954334 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r9bfq" event={"ID":"a41e843c-6800-4404-a7cb-adb5af6493f9","Type":"ContainerDied","Data":"217e19de94d34f70f58872827f7f70f2271b19e64ed1b5ece59c56ecbda0adbf"} Nov 03 14:57:51.954425 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.954363 6389 scope.go:122] "RemoveContainer" containerID="4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c" Nov 03 14:57:51.962994 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.962977 6389 scope.go:122] "RemoveContainer" containerID="4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c" Nov 03 14:57:51.963281 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:57:51.963260 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c\": container with ID starting with 4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c not found: ID does not exist" containerID="4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c" Nov 03 14:57:51.963327 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:51.963287 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c"} err="failed to get container status \"4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c\": rpc error: code = NotFound desc = could not find container \"4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c\": container with ID starting with 4b60b83868a2c14448197058a7ae1850bfecbfbd63f911d890a88185d9bbf85c not found: ID does not exist" Nov 03 14:57:52.009424 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:52.009391 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r9bfq"] Nov 03 14:57:52.022119 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:52.022089 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r9bfq"] Nov 03 14:57:52.617270 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:52.617240 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a41e843c-6800-4404-a7cb-adb5af6493f9" path="/var/lib/kubelet/pods/a41e843c-6800-4404-a7cb-adb5af6493f9/volumes" Nov 03 14:57:58.373294 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:57:58.373251 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:57:59.946942 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:57:59.946916 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-76bfcf8f7d-tzw9j" Nov 03 14:58:00.022208 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:00.022172 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-559f69b56-h5sc2"] Nov 03 14:58:03.676726 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.676686 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj"] Nov 03 14:58:03.677170 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.677097 6389 cpu_manager.go:414] "containerMap: removing container" podUID="a41e843c-6800-4404-a7cb-adb5af6493f9" containerName="payload-processing" Nov 03 14:58:03.677170 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.677109 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41e843c-6800-4404-a7cb-adb5af6493f9" containerName="payload-processing" Nov 03 14:58:03.677258 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.677198 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41e843c-6800-4404-a7cb-adb5af6493f9" containerName="payload-processing" Nov 03 14:58:03.680290 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.680271 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.703540 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.703513 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj"] Nov 03 14:58:03.714757 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.714730 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 03 14:58:03.714902 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.714828 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 14:58:03.714902 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.714851 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 14:58:03.714902 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.714878 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 14:58:03.715040 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.714923 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-48w7p\"" Nov 03 14:58:03.715040 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.714927 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 14:58:03.818348 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.818304 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-l5wbj\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.818348 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.818354 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ts9s\" (UniqueName: \"kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-kube-api-access-7ts9s\") pod \"payload-processing-external-model-5bcd7997-l5wbj\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.818718 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.818391 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-l5wbj\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.818718 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.818562 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-l5wbj\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.919837 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.919795 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-l5wbj\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.920015 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.919846 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-l5wbj\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.920015 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.919958 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ts9s\" (UniqueName: \"kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-kube-api-access-7ts9s\") pod \"payload-processing-external-model-5bcd7997-l5wbj\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.920015 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.919988 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-l5wbj\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.920491 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.920470 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-l5wbj\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.920597 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.920509 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-l5wbj\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.922531 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.922513 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-l5wbj\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.942160 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.942101 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ts9s\" (UniqueName: \"kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-kube-api-access-7ts9s\") pod \"payload-processing-external-model-5bcd7997-l5wbj\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:03.993498 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:03.993468 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:04.293654 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:04.293552 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj"] Nov 03 14:58:04.297790 ip-10-0-130-191 kubenswrapper[6389]: W1103 14:58:04.297759 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d8cf53_2514_4f64_b433_5edce1aa557d.slice/crio-6b6fac4403d420482f6029a078ca8f75e15f7b66117891516555764ad6989e98 WatchSource:0}: Error finding container 6b6fac4403d420482f6029a078ca8f75e15f7b66117891516555764ad6989e98: Status 404 returned error can't find the container with id 6b6fac4403d420482f6029a078ca8f75e15f7b66117891516555764ad6989e98 Nov 03 14:58:05.021961 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:05.021925 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" event={"ID":"37d8cf53-2514-4f64-b433-5edce1aa557d","Type":"ContainerStarted","Data":"b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5"} Nov 03 14:58:05.022497 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:05.021974 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" event={"ID":"37d8cf53-2514-4f64-b433-5edce1aa557d","Type":"ContainerStarted","Data":"6b6fac4403d420482f6029a078ca8f75e15f7b66117891516555764ad6989e98"} Nov 03 14:58:05.022497 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:05.022063 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:05.022813 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:05.022583 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" podUID="bca384f4-cf85-43ee-aa11-0ba7f2ea9d28" containerName="payload-pre-processing" containerID="cri-o://4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e" gracePeriod=25 Nov 03 14:58:05.067642 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:05.067561 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" podStartSLOduration=2.067541852 podStartE2EDuration="2.067541852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:58:05.065994271 +0000 UTC m=+2760.624630755" watchObservedRunningTime="2026-11-03 14:58:05.067541852 +0000 UTC m=+2760.626178337" Nov 03 14:58:05.330427 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:05.330394 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" Nov 03 14:58:05.437861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:05.437827 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-plugins-config-volume\") pod \"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28\" (UID: \"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28\") " Nov 03 14:58:05.437861 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:05.437869 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-kube-api-access-x8nrw\" (UniqueName: \"kubernetes.io/projected/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-kube-api-access-x8nrw\") pod \"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28\" (UID: \"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28\") " Nov 03 14:58:05.438348 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:05.438310 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-plugins-config-volume" pod "bca384f4-cf85-43ee-aa11-0ba7f2ea9d28" (UID: "bca384f4-cf85-43ee-aa11-0ba7f2ea9d28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:58:05.440202 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:05.440176 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-kube-api-access-x8nrw" pod "bca384f4-cf85-43ee-aa11-0ba7f2ea9d28" (UID: "bca384f4-cf85-43ee-aa11-0ba7f2ea9d28"). InnerVolumeSpecName "kube-api-access-x8nrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:58:05.538976 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:05.538940 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:58:05.538976 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:05.538974 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8nrw\" (UniqueName: \"kubernetes.io/projected/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28-kube-api-access-x8nrw\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:58:06.061643 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:06.059956 6389 generic.go:356] "Generic (PLEG): container finished" podID="bca384f4-cf85-43ee-aa11-0ba7f2ea9d28" containerID="4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e" exitCode=0 Nov 03 14:58:06.061643 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:06.060820 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" Nov 03 14:58:06.070006 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:06.069974 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" event={"ID":"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28","Type":"ContainerDied","Data":"4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e"} Nov 03 14:58:06.070279 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:06.070239 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-559f69b56-h5sc2" event={"ID":"bca384f4-cf85-43ee-aa11-0ba7f2ea9d28","Type":"ContainerDied","Data":"6040344cc4926d0828d22e92b05905396d1fd1e98c0d544a34ff505da1fd3bbd"} Nov 03 14:58:06.070477 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:06.070460 6389 scope.go:122] "RemoveContainer" containerID="4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e" Nov 03 14:58:06.107017 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:06.106998 6389 scope.go:122] "RemoveContainer" containerID="4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e" Nov 03 14:58:06.107375 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:58:06.107356 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e\": container with ID starting with 4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e not found: ID does not exist" containerID="4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e" Nov 03 14:58:06.107440 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:06.107386 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e"} err="failed to get container status \"4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e\": rpc error: code = NotFound desc = could not find container \"4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e\": container with ID starting with 4a9e3eb701a2b5ff10e175bc9597fc5b727a4eb141f0c9e59e87309fc8ee7f5e not found: ID does not exist" Nov 03 14:58:06.146503 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:06.146426 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-559f69b56-h5sc2"] Nov 03 14:58:06.169879 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:06.169833 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-559f69b56-h5sc2"] Nov 03 14:58:06.616416 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:06.616384 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca384f4-cf85-43ee-aa11-0ba7f2ea9d28" path="/var/lib/kubelet/pods/bca384f4-cf85-43ee-aa11-0ba7f2ea9d28/volumes" Nov 03 14:58:16.063786 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:58:16.063757 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:58:28.373598 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:58:28.373556 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:58:58.373202 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:58:58.373121 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:59:28.373382 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:59:28.373339 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 14:59:40.543978 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:40.543946 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj"] Nov 03 14:59:44.822759 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:44.822716 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:59:44.823430 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:44.823396 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:59:44.947556 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:44.947514 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:59:44.947860 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:44.947830 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:59:45.545095 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.545050 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" podUID="37d8cf53-2514-4f64-b433-5edce1aa557d" containerName="payload-processing" containerID="cri-o://b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5" gracePeriod=25 Nov 03 14:59:45.795513 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.795449 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:59:45.821697 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.821670 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-routing-overlay-volume\") pod \"37d8cf53-2514-4f64-b433-5edce1aa557d\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " Nov 03 14:59:45.821873 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.821779 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-provider-credentials\") pod \"37d8cf53-2514-4f64-b433-5edce1aa557d\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " Nov 03 14:59:45.821873 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.821816 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-plugins-config-volume\") pod \"37d8cf53-2514-4f64-b433-5edce1aa557d\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " Nov 03 14:59:45.821873 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.821853 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-kube-api-access-7ts9s\" (UniqueName: \"kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-kube-api-access-7ts9s\") pod \"37d8cf53-2514-4f64-b433-5edce1aa557d\" (UID: \"37d8cf53-2514-4f64-b433-5edce1aa557d\") " Nov 03 14:59:45.822104 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.822080 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-routing-overlay-volume" pod "37d8cf53-2514-4f64-b433-5edce1aa557d" (UID: "37d8cf53-2514-4f64-b433-5edce1aa557d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:59:45.822246 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.822224 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-plugins-config-volume" pod "37d8cf53-2514-4f64-b433-5edce1aa557d" (UID: "37d8cf53-2514-4f64-b433-5edce1aa557d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:59:45.824107 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.824081 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-provider-credentials" pod "37d8cf53-2514-4f64-b433-5edce1aa557d" (UID: "37d8cf53-2514-4f64-b433-5edce1aa557d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:59:45.824419 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.824163 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-kube-api-access-7ts9s" pod "37d8cf53-2514-4f64-b433-5edce1aa557d" (UID: "37d8cf53-2514-4f64-b433-5edce1aa557d"). InnerVolumeSpecName "kube-api-access-7ts9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:59:45.923189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.923151 6389 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-routing-overlay-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:59:45.923189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.923179 6389 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-provider-credentials\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:59:45.923189 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.923195 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d8cf53-2514-4f64-b433-5edce1aa557d-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:59:45.923427 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:45.923203 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ts9s\" (UniqueName: \"kubernetes.io/projected/37d8cf53-2514-4f64-b433-5edce1aa557d-kube-api-access-7ts9s\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 14:59:46.589239 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:46.589206 6389 generic.go:356] "Generic (PLEG): container finished" podID="37d8cf53-2514-4f64-b433-5edce1aa557d" containerID="b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5" exitCode=0 Nov 03 14:59:46.589526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:46.589256 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" Nov 03 14:59:46.589526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:46.589270 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" event={"ID":"37d8cf53-2514-4f64-b433-5edce1aa557d","Type":"ContainerDied","Data":"b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5"} Nov 03 14:59:46.589526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:46.589336 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj" event={"ID":"37d8cf53-2514-4f64-b433-5edce1aa557d","Type":"ContainerDied","Data":"6b6fac4403d420482f6029a078ca8f75e15f7b66117891516555764ad6989e98"} Nov 03 14:59:46.589526 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:46.589356 6389 scope.go:122] "RemoveContainer" containerID="b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5" Nov 03 14:59:46.598326 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:46.598306 6389 scope.go:122] "RemoveContainer" containerID="b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5" Nov 03 14:59:46.598584 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:59:46.598558 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5\": container with ID starting with b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5 not found: ID does not exist" containerID="b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5" Nov 03 14:59:46.598679 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:46.598599 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5"} err="failed to get container status \"b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5\": rpc error: code = NotFound desc = could not find container \"b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5\": container with ID starting with b581d8cda32b99d0bdd30b2ee0551d6e924a4be771a68583ea7a83d81f7964f5 not found: ID does not exist" Nov 03 14:59:46.648461 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:46.648430 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj"] Nov 03 14:59:46.661098 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:46.661058 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-l5wbj"] Nov 03 14:59:48.616223 ip-10-0-130-191 kubenswrapper[6389]: I1103 14:59:48.616191 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d8cf53-2514-4f64-b433-5edce1aa557d" path="/var/lib/kubelet/pods/37d8cf53-2514-4f64-b433-5edce1aa557d/volumes" Nov 03 14:59:58.373083 ip-10-0-130-191 kubenswrapper[6389]: E1103 14:59:58.373040 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:00:14.817796 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:00:14.817751 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:00:14.818221 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:00:14.817832 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:00:14.955634 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:00:14.954731 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:00:14.955634 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:00:14.954797 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:00:28.374053 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:00:28.374014 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:00:58.373024 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:00:58.372992 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:01:28.373271 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:01:28.373230 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:01:58.373467 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:01:58.373367 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:02:28.372684 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:02:28.372643 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:02:58.373496 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:02:58.373458 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:03:14.824632 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:03:14.822725 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:03:14.824632 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:03:14.822809 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:03:14.952775 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:03:14.952736 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:03:14.953099 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:03:14.953054 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:03:28.390057 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:03:28.390001 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:03:58.412276 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:03:58.412231 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:03:59.820787 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:03:59.820753 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:03:59.821359 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:03:59.820823 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:04:00.408303 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.408274 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5"] Nov 03 15:04:00.408757 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.408735 6389 cpu_manager.go:414] "containerMap: removing container" podUID="bca384f4-cf85-43ee-aa11-0ba7f2ea9d28" containerName="payload-pre-processing" Nov 03 15:04:00.408757 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.408758 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca384f4-cf85-43ee-aa11-0ba7f2ea9d28" containerName="payload-pre-processing" Nov 03 15:04:00.408912 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.408793 6389 cpu_manager.go:414] "containerMap: removing container" podUID="37d8cf53-2514-4f64-b433-5edce1aa557d" containerName="payload-processing" Nov 03 15:04:00.408912 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.408800 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d8cf53-2514-4f64-b433-5edce1aa557d" containerName="payload-processing" Nov 03 15:04:00.408912 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.408863 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca384f4-cf85-43ee-aa11-0ba7f2ea9d28" containerName="payload-pre-processing" Nov 03 15:04:00.408912 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.408874 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d8cf53-2514-4f64-b433-5edce1aa557d" containerName="payload-processing" Nov 03 15:04:00.411722 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.411702 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.428596 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.428577 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ecd9e5-openshift-default-dockercfg-tcsx8\"" Nov 03 15:04:00.441574 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.441545 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5"] Nov 03 15:04:00.606347 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.606307 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-credential-socket\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.606549 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.606357 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d32c7b-9523-4762-8705-a5918ece62c8-istio-podinfo\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.606549 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.606487 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-data\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.606689 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.606549 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-envoy\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.606689 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.606648 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-socket\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.606798 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.606747 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-istio-token\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.606798 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.606776 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-certs\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.606908 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.606817 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbffk\" (UniqueName: \"kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-kube-api-access-cbffk\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.606908 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.606867 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d32c7b-9523-4762-8705-a5918ece62c8-istiod-ca-cert\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708165 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708129 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-data\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708343 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708182 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-envoy\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708343 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708213 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-socket\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708437 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708374 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-istio-token\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708437 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708414 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-certs\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708549 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708455 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbffk\" (UniqueName: \"kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-kube-api-access-cbffk\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708549 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708485 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d32c7b-9523-4762-8705-a5918ece62c8-istiod-ca-cert\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708689 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708573 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-credential-socket\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708689 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708582 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-data\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708689 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708592 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-socket\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708689 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708597 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d32c7b-9523-4762-8705-a5918ece62c8-istio-podinfo\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708882 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708840 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-certs\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.708980 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.708964 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-credential-socket\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.709237 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.709214 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d32c7b-9523-4762-8705-a5918ece62c8-istiod-ca-cert\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.710775 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.710750 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-envoy\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.711328 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.711311 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d32c7b-9523-4762-8705-a5918ece62c8-istio-podinfo\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.730559 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.730528 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-istio-token\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:00.731041 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:00.731019 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbffk\" (UniqueName: \"kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-kube-api-access-cbffk\") pod \"e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:01.025324 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:01.025250 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:01.345812 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:01.345782 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5"] Nov 03 15:04:01.348494 ip-10-0-130-191 kubenswrapper[6389]: W1103 15:04:01.348468 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d32c7b_9523_4762_8705_a5918ece62c8.slice/crio-4c65d06d6f4e5443b30abfe726a852bfb4b6772954e1cfd11806305efe08262d WatchSource:0}: Error finding container 4c65d06d6f4e5443b30abfe726a852bfb4b6772954e1cfd11806305efe08262d: Status 404 returned error can't find the container with id 4c65d06d6f4e5443b30abfe726a852bfb4b6772954e1cfd11806305efe08262d Nov 03 15:04:01.350747 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:01.350713 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 15:04:01.350833 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:01.350783 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 15:04:01.350833 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:01.350809 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 15:04:01.929360 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:01.929323 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" event={"ID":"41d32c7b-9523-4762-8705-a5918ece62c8","Type":"ContainerStarted","Data":"f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25"} Nov 03 15:04:01.929544 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:01.929375 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" event={"ID":"41d32c7b-9523-4762-8705-a5918ece62c8","Type":"ContainerStarted","Data":"4c65d06d6f4e5443b30abfe726a852bfb4b6772954e1cfd11806305efe08262d"} Nov 03 15:04:02.016400 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:02.016341 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" podStartSLOduration=2.016328595 podStartE2EDuration="2.016328595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:04:02.015300023 +0000 UTC m=+3117.573936507" watchObservedRunningTime="2026-11-03 15:04:02.016328595 +0000 UTC m=+3117.574965120" Nov 03 15:04:02.025584 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:02.025560 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:02.030638 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:02.030602 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:02.934603 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:02.934568 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:02.935490 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:02.935471 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:06.156957 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.156923 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x"] Nov 03 15:04:06.160714 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.160690 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.192452 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.192420 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ecd9e5-serving-cert\"" Nov 03 15:04:06.194178 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.193969 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x"] Nov 03 15:04:06.259745 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.259706 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqsns\" (UniqueName: \"kubernetes.io/projected/2e617043-6636-450c-bd00-7032b2049ecc-kube-api-access-nqsns\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.259919 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.259782 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.260018 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.259992 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.260062 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.260051 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e617043-6636-450c-bd00-7032b2049ecc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.360582 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.360551 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqsns\" (UniqueName: \"kubernetes.io/projected/2e617043-6636-450c-bd00-7032b2049ecc-kube-api-access-nqsns\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.360798 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.360643 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.360798 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.360728 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.360798 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.360766 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e617043-6636-450c-bd00-7032b2049ecc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.361795 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.361763 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e617043-6636-450c-bd00-7032b2049ecc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.364027 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.363994 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.364162 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.364077 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.382177 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.382158 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqsns\" (UniqueName: \"kubernetes.io/projected/2e617043-6636-450c-bd00-7032b2049ecc-kube-api-access-nqsns\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.481814 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.481776 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:06.817128 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.817107 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x"] Nov 03 15:04:06.818774 ip-10-0-130-191 kubenswrapper[6389]: W1103 15:04:06.818746 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e617043_6636_450c_bd00_7032b2049ecc.slice/crio-f0242cf0a4cb7001b34deae64f10596ded6a7883119f49adc235198d95c880bf WatchSource:0}: Error finding container f0242cf0a4cb7001b34deae64f10596ded6a7883119f49adc235198d95c880bf: Status 404 returned error can't find the container with id f0242cf0a4cb7001b34deae64f10596ded6a7883119f49adc235198d95c880bf Nov 03 15:04:06.820276 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.820256 6389 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:04:06.953050 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:06.953013 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" event={"ID":"2e617043-6636-450c-bd00-7032b2049ecc","Type":"ContainerStarted","Data":"f0242cf0a4cb7001b34deae64f10596ded6a7883119f49adc235198d95c880bf"} Nov 03 15:04:07.959001 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:07.958960 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" event={"ID":"2e617043-6636-450c-bd00-7032b2049ecc","Type":"ContainerStarted","Data":"b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747"} Nov 03 15:04:07.959452 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:07.959065 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:08.006452 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:08.006395 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" podStartSLOduration=1.4873617160000001 podStartE2EDuration="2.00637752s" totalImagesPullingTime="519.015804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:04:08.004888339 +0000 UTC m=+3123.563524833" watchObservedRunningTime="2026-11-03 15:04:08.00637752 +0000 UTC m=+3123.565014003" Nov 03 15:04:13.969658 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:13.969596 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:04:18.977777 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:18.977732 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:04:18.978220 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:18.977801 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:04:20.016470 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:20.016437 6389 generic.go:356] "Generic (PLEG): container finished" podID="41d32c7b-9523-4762-8705-a5918ece62c8" containerID="f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25" exitCode=0 Nov 03 15:04:20.017023 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:20.016512 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" event={"ID":"41d32c7b-9523-4762-8705-a5918ece62c8","Type":"ContainerDied","Data":"f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25"} Nov 03 15:04:20.017023 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:20.016969 6389 scope.go:122] "RemoveContainer" containerID="f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25" Nov 03 15:04:20.018210 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:20.018179 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 15:04:20.018288 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:20.018245 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 15:04:20.018288 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:20.018276 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 15:04:21.041534 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:21.041491 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" event={"ID":"41d32c7b-9523-4762-8705-a5918ece62c8","Type":"ContainerStarted","Data":"4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe"} Nov 03 15:04:22.027984 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:22.027949 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:22.034770 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:22.034735 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 15:04:22.035196 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:22.035165 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 15:04:23.030121 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:23.030084 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 15:04:23.030803 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:23.030759 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 15:04:24.034071 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:24.034033 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 15:04:24.034734 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:24.034700 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 15:04:25.031184 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:25.031134 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 15:04:25.031509 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:25.031468 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 15:04:26.026559 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:26.026516 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 15:04:26.026995 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:26.026594 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 15:04:27.029880 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:27.029843 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:27.030282 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:27.030145 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:27.030790 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:04:27.030769 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:04:28.373427 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:04:28.373389 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:04:58.373397 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:04:58.373320 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:05:11.742421 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:11.742380 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x"] Nov 03 15:05:11.743049 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:11.742706 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" podUID="2e617043-6636-450c-bd00-7032b2049ecc" containerName="maas-api" containerID="cri-o://b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747" gracePeriod=30 Nov 03 15:05:12.242431 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.242406 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:05:12.298940 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.298853 6389 generic.go:356] "Generic (PLEG): container finished" podID="2e617043-6636-450c-bd00-7032b2049ecc" containerID="b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747" exitCode=0 Nov 03 15:05:12.298940 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.298911 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" Nov 03 15:05:12.299194 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.298933 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" event={"ID":"2e617043-6636-450c-bd00-7032b2049ecc","Type":"ContainerDied","Data":"b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747"} Nov 03 15:05:12.299194 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.298996 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x" event={"ID":"2e617043-6636-450c-bd00-7032b2049ecc","Type":"ContainerDied","Data":"f0242cf0a4cb7001b34deae64f10596ded6a7883119f49adc235198d95c880bf"} Nov 03 15:05:12.299194 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.299028 6389 scope.go:122] "RemoveContainer" containerID="b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747" Nov 03 15:05:12.308360 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.308340 6389 scope.go:122] "RemoveContainer" containerID="b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747" Nov 03 15:05:12.308721 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:05:12.308694 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747\": container with ID starting with b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747 not found: ID does not exist" containerID="b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747" Nov 03 15:05:12.308805 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.308733 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747"} err="failed to get container status \"b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747\": rpc error: code = NotFound desc = could not find container \"b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747\": container with ID starting with b7eb9b0e07174ac5372484e6e0b5920e375684e890cfa3eb54eeea46044db747 not found: ID does not exist" Nov 03 15:05:12.403195 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.403081 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-metrics-certs\") pod \"2e617043-6636-450c-bd00-7032b2049ecc\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " Nov 03 15:05:12.403459 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.403429 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e617043-6636-450c-bd00-7032b2049ecc-kube-api-access-nqsns\" (UniqueName: \"kubernetes.io/projected/2e617043-6636-450c-bd00-7032b2049ecc-kube-api-access-nqsns\") pod \"2e617043-6636-450c-bd00-7032b2049ecc\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " Nov 03 15:05:12.403598 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.403545 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-maas-api-tls\") pod \"2e617043-6636-450c-bd00-7032b2049ecc\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " Nov 03 15:05:12.403711 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.403602 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e617043-6636-450c-bd00-7032b2049ecc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e617043-6636-450c-bd00-7032b2049ecc-openshift-service-ca\") pod \"2e617043-6636-450c-bd00-7032b2049ecc\" (UID: \"2e617043-6636-450c-bd00-7032b2049ecc\") " Nov 03 15:05:12.404219 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.404196 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e617043-6636-450c-bd00-7032b2049ecc-openshift-service-ca" pod "2e617043-6636-450c-bd00-7032b2049ecc" (UID: "2e617043-6636-450c-bd00-7032b2049ecc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:05:12.405453 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.405433 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-metrics-certs" pod "2e617043-6636-450c-bd00-7032b2049ecc" (UID: "2e617043-6636-450c-bd00-7032b2049ecc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:05:12.405560 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.405542 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e617043-6636-450c-bd00-7032b2049ecc-kube-api-access-nqsns" pod "2e617043-6636-450c-bd00-7032b2049ecc" (UID: "2e617043-6636-450c-bd00-7032b2049ecc"). InnerVolumeSpecName "kube-api-access-nqsns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:05:12.405996 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.405972 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-maas-api-tls" pod "2e617043-6636-450c-bd00-7032b2049ecc" (UID: "2e617043-6636-450c-bd00-7032b2049ecc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:05:12.505355 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.505322 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:05:12.505355 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.505350 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqsns\" (UniqueName: \"kubernetes.io/projected/2e617043-6636-450c-bd00-7032b2049ecc-kube-api-access-nqsns\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:05:12.505355 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.505359 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e617043-6636-450c-bd00-7032b2049ecc-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:05:12.505591 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.505368 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e617043-6636-450c-bd00-7032b2049ecc-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:05:12.646725 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.646688 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x"] Nov 03 15:05:12.659854 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:12.659823 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9rl2x"] Nov 03 15:05:14.639965 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:14.638923 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e617043-6636-450c-bd00-7032b2049ecc" path="/var/lib/kubelet/pods/2e617043-6636-450c-bd00-7032b2049ecc/volumes" Nov 03 15:05:28.373520 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:05:28.373483 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:05:43.944327 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:43.944290 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j"] Nov 03 15:05:43.944951 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:43.944923 6389 cpu_manager.go:414] "containerMap: removing container" podUID="2e617043-6636-450c-bd00-7032b2049ecc" containerName="maas-api" Nov 03 15:05:43.944951 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:43.944941 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e617043-6636-450c-bd00-7032b2049ecc" containerName="maas-api" Nov 03 15:05:43.945084 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:43.945046 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e617043-6636-450c-bd00-7032b2049ecc" containerName="maas-api" Nov 03 15:05:43.948185 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:43.948165 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:43.965844 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:43.965818 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ecd9e5-serving-cert\"" Nov 03 15:05:43.980317 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:43.980269 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j"] Nov 03 15:05:44.107892 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.107853 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8f7w\" (UniqueName: \"kubernetes.io/projected/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-kube-api-access-w8f7w\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.108097 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.107907 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.108097 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.108019 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.108097 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.108070 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.209549 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.209457 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.209549 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.209539 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8f7w\" (UniqueName: \"kubernetes.io/projected/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-kube-api-access-w8f7w\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.209799 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.209561 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.209799 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.209631 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.210450 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.210340 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.212437 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.212414 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.212536 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.212429 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.232443 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.232413 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8f7w\" (UniqueName: \"kubernetes.io/projected/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-kube-api-access-w8f7w\") pod \"maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.261328 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.261293 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:44.577282 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:44.577206 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j"] Nov 03 15:05:44.580183 ip-10-0-130-191 kubenswrapper[6389]: W1103 15:05:44.580156 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f17cba_aa2d_42c1_bd88_ba0f403d6279.slice/crio-832bad874c75be76c951dff28818cf8f423334f154ec9aa37d2b91434e5db088 WatchSource:0}: Error finding container 832bad874c75be76c951dff28818cf8f423334f154ec9aa37d2b91434e5db088: Status 404 returned error can't find the container with id 832bad874c75be76c951dff28818cf8f423334f154ec9aa37d2b91434e5db088 Nov 03 15:05:45.481056 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:45.481021 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" event={"ID":"c6f17cba-aa2d-42c1-bd88-ba0f403d6279","Type":"ContainerStarted","Data":"5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083"} Nov 03 15:05:45.481470 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:45.481080 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" event={"ID":"c6f17cba-aa2d-42c1-bd88-ba0f403d6279","Type":"ContainerStarted","Data":"832bad874c75be76c951dff28818cf8f423334f154ec9aa37d2b91434e5db088"} Nov 03 15:05:45.481470 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:45.481110 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:45.529692 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:45.529630 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" podStartSLOduration=2.20107211 podStartE2EDuration="2.529601626s" totalImagesPullingTime="328.529516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:05:45.526926362 +0000 UTC m=+3221.085563035" watchObservedRunningTime="2026-11-03 15:05:45.529601626 +0000 UTC m=+3221.088238109" Nov 03 15:05:51.492744 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:05:51.492718 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:05:58.374166 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:05:58.374130 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:06:07.088331 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.088294 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j"] Nov 03 15:06:07.088901 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.088542 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" podUID="c6f17cba-aa2d-42c1-bd88-ba0f403d6279" containerName="maas-api" containerID="cri-o://5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083" gracePeriod=30 Nov 03 15:06:07.353396 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.353312 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:06:07.472174 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.472136 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-openshift-service-ca\") pod \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " Nov 03 15:06:07.472382 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.472189 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-maas-api-tls\") pod \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " Nov 03 15:06:07.472382 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.472283 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-metrics-certs\") pod \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " Nov 03 15:06:07.472382 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.472344 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-kube-api-access-w8f7w\" (UniqueName: \"kubernetes.io/projected/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-kube-api-access-w8f7w\") pod \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\" (UID: \"c6f17cba-aa2d-42c1-bd88-ba0f403d6279\") " Nov 03 15:06:07.472659 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.472588 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-openshift-service-ca" pod "c6f17cba-aa2d-42c1-bd88-ba0f403d6279" (UID: "c6f17cba-aa2d-42c1-bd88-ba0f403d6279"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:06:07.474646 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.474603 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-metrics-certs" pod "c6f17cba-aa2d-42c1-bd88-ba0f403d6279" (UID: "c6f17cba-aa2d-42c1-bd88-ba0f403d6279"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:06:07.474750 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.474718 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-kube-api-access-w8f7w" pod "c6f17cba-aa2d-42c1-bd88-ba0f403d6279" (UID: "c6f17cba-aa2d-42c1-bd88-ba0f403d6279"). InnerVolumeSpecName "kube-api-access-w8f7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:06:07.474750 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.474728 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-maas-api-tls" pod "c6f17cba-aa2d-42c1-bd88-ba0f403d6279" (UID: "c6f17cba-aa2d-42c1-bd88-ba0f403d6279"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:06:07.573354 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.573316 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:07.573354 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.573348 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8f7w\" (UniqueName: \"kubernetes.io/projected/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-kube-api-access-w8f7w\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:07.573354 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.573358 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:07.573354 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.573367 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6f17cba-aa2d-42c1-bd88-ba0f403d6279-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:07.591495 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.591465 6389 generic.go:356] "Generic (PLEG): container finished" podID="c6f17cba-aa2d-42c1-bd88-ba0f403d6279" containerID="5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083" exitCode=0 Nov 03 15:06:07.591687 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.591545 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" Nov 03 15:06:07.591687 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.591571 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" event={"ID":"c6f17cba-aa2d-42c1-bd88-ba0f403d6279","Type":"ContainerDied","Data":"5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083"} Nov 03 15:06:07.591800 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.591719 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j" event={"ID":"c6f17cba-aa2d-42c1-bd88-ba0f403d6279","Type":"ContainerDied","Data":"832bad874c75be76c951dff28818cf8f423334f154ec9aa37d2b91434e5db088"} Nov 03 15:06:07.591800 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.591754 6389 scope.go:122] "RemoveContainer" containerID="5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083" Nov 03 15:06:07.601524 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.601509 6389 scope.go:122] "RemoveContainer" containerID="5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083" Nov 03 15:06:07.601811 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:06:07.601794 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083\": container with ID starting with 5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083 not found: ID does not exist" containerID="5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083" Nov 03 15:06:07.601856 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.601820 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083"} err="failed to get container status \"5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083\": rpc error: code = NotFound desc = could not find container \"5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083\": container with ID starting with 5b9590a5c1d8dc70c32236df58cb23e73c4742722fd24bff0d3e4e8445e79083 not found: ID does not exist" Nov 03 15:06:07.647414 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.647385 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j"] Nov 03 15:06:07.649898 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:07.649878 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ecd9e5-559bf956f9-9jw2j"] Nov 03 15:06:08.310956 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:08.310926 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5"] Nov 03 15:06:08.311415 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:08.311177 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" containerID="cri-o://4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe" gracePeriod=30 Nov 03 15:06:08.616195 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:08.616119 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f17cba-aa2d-42c1-bd88-ba0f403d6279" path="/var/lib/kubelet/pods/c6f17cba-aa2d-42c1-bd88-ba0f403d6279/volumes" Nov 03 15:06:12.032279 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:12.032234 6389 patch_prober.go:29] interesting pod/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:06:12.032686 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:12.032297 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:06:13.563075 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.563052 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:06:13.620146 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.620054 6389 generic.go:356] "Generic (PLEG): container finished" podID="41d32c7b-9523-4762-8705-a5918ece62c8" containerID="4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe" exitCode=0 Nov 03 15:06:13.620146 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.620095 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" Nov 03 15:06:13.620146 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.620123 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" event={"ID":"41d32c7b-9523-4762-8705-a5918ece62c8","Type":"ContainerDied","Data":"4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe"} Nov 03 15:06:13.620404 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.620174 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5" event={"ID":"41d32c7b-9523-4762-8705-a5918ece62c8","Type":"ContainerDied","Data":"4c65d06d6f4e5443b30abfe726a852bfb4b6772954e1cfd11806305efe08262d"} Nov 03 15:06:13.620404 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.620197 6389 scope.go:122] "RemoveContainer" containerID="4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe" Nov 03 15:06:13.631603 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.631582 6389 scope.go:122] "RemoveContainer" containerID="f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25" Nov 03 15:06:13.635010 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.634970 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-socket\") pod \"41d32c7b-9523-4762-8705-a5918ece62c8\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " Nov 03 15:06:13.635210 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.635048 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-credential-socket\") pod \"41d32c7b-9523-4762-8705-a5918ece62c8\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " Nov 03 15:06:13.635210 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.635085 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-kube-api-access-cbffk\" (UniqueName: \"kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-kube-api-access-cbffk\") pod \"41d32c7b-9523-4762-8705-a5918ece62c8\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " Nov 03 15:06:13.635210 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.635119 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d32c7b-9523-4762-8705-a5918ece62c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d32c7b-9523-4762-8705-a5918ece62c8-istiod-ca-cert\") pod \"41d32c7b-9523-4762-8705-a5918ece62c8\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " Nov 03 15:06:13.635210 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.635178 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-certs\") pod \"41d32c7b-9523-4762-8705-a5918ece62c8\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " Nov 03 15:06:13.635508 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.635240 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-socket" pod "41d32c7b-9523-4762-8705-a5918ece62c8" (UID: "41d32c7b-9523-4762-8705-a5918ece62c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:06:13.635508 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.635441 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-certs" pod "41d32c7b-9523-4762-8705-a5918ece62c8" (UID: "41d32c7b-9523-4762-8705-a5918ece62c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:06:13.635654 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.635634 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-envoy\") pod \"41d32c7b-9523-4762-8705-a5918ece62c8\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " Nov 03 15:06:13.635718 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.635658 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-credential-socket" pod "41d32c7b-9523-4762-8705-a5918ece62c8" (UID: "41d32c7b-9523-4762-8705-a5918ece62c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:06:13.635771 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.635712 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-data\") pod \"41d32c7b-9523-4762-8705-a5918ece62c8\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " Nov 03 15:06:13.635771 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.635752 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41d32c7b-9523-4762-8705-a5918ece62c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d32c7b-9523-4762-8705-a5918ece62c8-istio-podinfo\") pod \"41d32c7b-9523-4762-8705-a5918ece62c8\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " Nov 03 15:06:13.635888 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.635870 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-istio-token\" (UniqueName: \"kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-istio-token\") pod \"41d32c7b-9523-4762-8705-a5918ece62c8\" (UID: \"41d32c7b-9523-4762-8705-a5918ece62c8\") " Nov 03 15:06:13.636233 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.636074 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-data" pod "41d32c7b-9523-4762-8705-a5918ece62c8" (UID: "41d32c7b-9523-4762-8705-a5918ece62c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:06:13.636333 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.636255 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d32c7b-9523-4762-8705-a5918ece62c8-istiod-ca-cert" pod "41d32c7b-9523-4762-8705-a5918ece62c8" (UID: "41d32c7b-9523-4762-8705-a5918ece62c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:06:13.636333 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.636279 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:13.636333 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.636296 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:13.636333 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.636310 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:13.636333 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.636323 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:13.638910 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.638884 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-envoy" pod "41d32c7b-9523-4762-8705-a5918ece62c8" (UID: "41d32c7b-9523-4762-8705-a5918ece62c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:06:13.639508 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.639477 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-istio-token" pod "41d32c7b-9523-4762-8705-a5918ece62c8" (UID: "41d32c7b-9523-4762-8705-a5918ece62c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:06:13.640480 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.640455 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41d32c7b-9523-4762-8705-a5918ece62c8-istio-podinfo" pod "41d32c7b-9523-4762-8705-a5918ece62c8" (UID: "41d32c7b-9523-4762-8705-a5918ece62c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:06:13.642710 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.642688 6389 scope.go:122] "RemoveContainer" containerID="4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe" Nov 03 15:06:13.643033 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.643012 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-kube-api-access-cbffk" pod "41d32c7b-9523-4762-8705-a5918ece62c8" (UID: "41d32c7b-9523-4762-8705-a5918ece62c8"). InnerVolumeSpecName "kube-api-access-cbffk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:06:13.643033 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:06:13.643018 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe\": container with ID starting with 4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe not found: ID does not exist" containerID="4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe" Nov 03 15:06:13.643148 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.643050 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe"} err="failed to get container status \"4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe\": rpc error: code = NotFound desc = could not find container \"4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe\": container with ID starting with 4435be27d0e589855a46e2fe41834a10251128d343dbc4fe6bacf3f9efcaeefe not found: ID does not exist" Nov 03 15:06:13.643148 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.643067 6389 scope.go:122] "RemoveContainer" containerID="f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25" Nov 03 15:06:13.643350 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:06:13.643327 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25\": container with ID starting with f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25 not found: ID does not exist" containerID="f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25" Nov 03 15:06:13.643409 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.643358 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25"} err="failed to get container status \"f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25\": rpc error: code = NotFound desc = could not find container \"f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25\": container with ID starting with f5b4edf321cab94000bee9490188bd2962673cf0a706c35d1beab1400c0dac25 not found: ID does not exist" Nov 03 15:06:13.737298 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.737270 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d32c7b-9523-4762-8705-a5918ece62c8-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:13.737298 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.737295 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d32c7b-9523-4762-8705-a5918ece62c8-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:13.737298 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.737304 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:13.737565 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.737315 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbffk\" (UniqueName: \"kubernetes.io/projected/41d32c7b-9523-4762-8705-a5918ece62c8-kube-api-access-cbffk\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:13.737565 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.737325 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d32c7b-9523-4762-8705-a5918ece62c8-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:06:13.969178 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.969147 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5"] Nov 03 15:06:13.976155 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:13.976136 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ecd9e5-openshift-default-549cb596ccclcp5"] Nov 03 15:06:14.616809 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:06:14.616774 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" path="/var/lib/kubelet/pods/41d32c7b-9523-4762-8705-a5918ece62c8/volumes" Nov 03 15:06:28.373485 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:06:28.373449 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:06:58.373339 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:06:58.373301 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:07:28.373454 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:07:28.373418 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:07:58.374110 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:07:58.374018 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:08:25.320487 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.320453 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7"] Nov 03 15:08:25.320888 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.320835 6389 cpu_manager.go:414] "containerMap: removing container" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" Nov 03 15:08:25.320888 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.320846 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" Nov 03 15:08:25.320888 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.320873 6389 cpu_manager.go:414] "containerMap: removing container" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" Nov 03 15:08:25.320888 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.320878 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" Nov 03 15:08:25.321024 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.320893 6389 cpu_manager.go:414] "containerMap: removing container" podUID="c6f17cba-aa2d-42c1-bd88-ba0f403d6279" containerName="maas-api" Nov 03 15:08:25.321024 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.320898 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f17cba-aa2d-42c1-bd88-ba0f403d6279" containerName="maas-api" Nov 03 15:08:25.321024 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.320953 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" Nov 03 15:08:25.321024 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.320964 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f17cba-aa2d-42c1-bd88-ba0f403d6279" containerName="maas-api" Nov 03 15:08:25.324058 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.324040 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.354404 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.354374 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7"] Nov 03 15:08:25.430963 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.430924 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5591b5-8964-4d8b-b94d-934ff325b6aa-openshift-service-ca\") pod \"maas-api-84b5dd6748-qqjc7\" (UID: \"3a5591b5-8964-4d8b-b94d-934ff325b6aa\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.431131 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.430978 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5591b5-8964-4d8b-b94d-934ff325b6aa-metrics-certs\") pod \"maas-api-84b5dd6748-qqjc7\" (UID: \"3a5591b5-8964-4d8b-b94d-934ff325b6aa\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.431131 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.431057 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5591b5-8964-4d8b-b94d-934ff325b6aa-maas-api-tls\") pod \"maas-api-84b5dd6748-qqjc7\" (UID: \"3a5591b5-8964-4d8b-b94d-934ff325b6aa\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.431131 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.431085 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68j8j\" (UniqueName: \"kubernetes.io/projected/3a5591b5-8964-4d8b-b94d-934ff325b6aa-kube-api-access-68j8j\") pod \"maas-api-84b5dd6748-qqjc7\" (UID: \"3a5591b5-8964-4d8b-b94d-934ff325b6aa\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.532517 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.532480 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5591b5-8964-4d8b-b94d-934ff325b6aa-maas-api-tls\") pod \"maas-api-84b5dd6748-qqjc7\" (UID: \"3a5591b5-8964-4d8b-b94d-934ff325b6aa\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.532517 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.532517 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68j8j\" (UniqueName: \"kubernetes.io/projected/3a5591b5-8964-4d8b-b94d-934ff325b6aa-kube-api-access-68j8j\") pod \"maas-api-84b5dd6748-qqjc7\" (UID: \"3a5591b5-8964-4d8b-b94d-934ff325b6aa\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.532787 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.532656 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5591b5-8964-4d8b-b94d-934ff325b6aa-openshift-service-ca\") pod \"maas-api-84b5dd6748-qqjc7\" (UID: \"3a5591b5-8964-4d8b-b94d-934ff325b6aa\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.532787 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.532700 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5591b5-8964-4d8b-b94d-934ff325b6aa-metrics-certs\") pod \"maas-api-84b5dd6748-qqjc7\" (UID: \"3a5591b5-8964-4d8b-b94d-934ff325b6aa\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.533407 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.533381 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5591b5-8964-4d8b-b94d-934ff325b6aa-openshift-service-ca\") pod \"maas-api-84b5dd6748-qqjc7\" (UID: \"3a5591b5-8964-4d8b-b94d-934ff325b6aa\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.535132 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.535106 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5591b5-8964-4d8b-b94d-934ff325b6aa-maas-api-tls\") pod \"maas-api-84b5dd6748-qqjc7\" (UID: \"3a5591b5-8964-4d8b-b94d-934ff325b6aa\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.535362 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.535342 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5591b5-8964-4d8b-b94d-934ff325b6aa-metrics-certs\") pod \"maas-api-84b5dd6748-qqjc7\" (UID: \"3a5591b5-8964-4d8b-b94d-934ff325b6aa\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.556348 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.556319 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68j8j\" (UniqueName: \"kubernetes.io/projected/3a5591b5-8964-4d8b-b94d-934ff325b6aa-kube-api-access-68j8j\") pod \"maas-api-84b5dd6748-qqjc7\" (UID: \"3a5591b5-8964-4d8b-b94d-934ff325b6aa\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.638388 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.638298 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:25.941890 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:25.941865 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7"] Nov 03 15:08:25.944724 ip-10-0-130-191 kubenswrapper[6389]: W1103 15:08:25.944691 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5591b5_8964_4d8b_b94d_934ff325b6aa.slice/crio-11d6c53c6587246d5ac3d6153b3ed1d97eee950474d13d2b43fcdb38ec6ec651 WatchSource:0}: Error finding container 11d6c53c6587246d5ac3d6153b3ed1d97eee950474d13d2b43fcdb38ec6ec651: Status 404 returned error can't find the container with id 11d6c53c6587246d5ac3d6153b3ed1d97eee950474d13d2b43fcdb38ec6ec651 Nov 03 15:08:26.271814 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:26.271734 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" event={"ID":"3a5591b5-8964-4d8b-b94d-934ff325b6aa","Type":"ContainerStarted","Data":"11d6c53c6587246d5ac3d6153b3ed1d97eee950474d13d2b43fcdb38ec6ec651"} Nov 03 15:08:27.277356 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:27.277324 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" event={"ID":"3a5591b5-8964-4d8b-b94d-934ff325b6aa","Type":"ContainerStarted","Data":"647e3618b9e89fc67c2fb00987f633de951a1bdcda0005ecfbe9f5ea254c55f9"} Nov 03 15:08:27.277809 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:27.277405 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:27.324419 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:27.324362 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" podStartSLOduration=1.946314879 podStartE2EDuration="2.324348434s" totalImagesPullingTime="378.033555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:27.32410758 +0000 UTC m=+3382.882744062" watchObservedRunningTime="2026-11-03 15:08:27.324348434 +0000 UTC m=+3382.882984918" Nov 03 15:08:28.374411 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:08:28.374368 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:08:33.288105 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:33.288073 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-qqjc7" Nov 03 15:08:33.405205 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:33.405171 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5757765976-lklsf"] Nov 03 15:08:33.405433 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:33.405406 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" podUID="466189f1-5bf9-4bb4-8eee-874a7407bc8a" containerName="maas-api" containerID="cri-o://91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2" gracePeriod=30 Nov 03 15:08:34.664951 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.664924 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 15:08:34.721775 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.721746 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/466189f1-5bf9-4bb4-8eee-874a7407bc8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466189f1-5bf9-4bb4-8eee-874a7407bc8a-openshift-service-ca\") pod \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " Nov 03 15:08:34.721967 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.721783 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-maas-api-tls\") pod \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " Nov 03 15:08:34.721967 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.721827 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466189f1-5bf9-4bb4-8eee-874a7407bc8a-kube-api-access-vbvt7\" (UniqueName: \"kubernetes.io/projected/466189f1-5bf9-4bb4-8eee-874a7407bc8a-kube-api-access-vbvt7\") pod \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " Nov 03 15:08:34.722099 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.721968 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-metrics-certs\") pod \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\" (UID: \"466189f1-5bf9-4bb4-8eee-874a7407bc8a\") " Nov 03 15:08:34.722188 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.722166 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466189f1-5bf9-4bb4-8eee-874a7407bc8a-openshift-service-ca" pod "466189f1-5bf9-4bb4-8eee-874a7407bc8a" (UID: "466189f1-5bf9-4bb4-8eee-874a7407bc8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:08:34.722315 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.722294 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466189f1-5bf9-4bb4-8eee-874a7407bc8a-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:08:34.724216 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.724189 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466189f1-5bf9-4bb4-8eee-874a7407bc8a-kube-api-access-vbvt7" pod "466189f1-5bf9-4bb4-8eee-874a7407bc8a" (UID: "466189f1-5bf9-4bb4-8eee-874a7407bc8a"). InnerVolumeSpecName "kube-api-access-vbvt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:08:34.724442 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.724415 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-metrics-certs" pod "466189f1-5bf9-4bb4-8eee-874a7407bc8a" (UID: "466189f1-5bf9-4bb4-8eee-874a7407bc8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:08:34.724534 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.724447 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-maas-api-tls" pod "466189f1-5bf9-4bb4-8eee-874a7407bc8a" (UID: "466189f1-5bf9-4bb4-8eee-874a7407bc8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:08:34.822948 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.822874 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:08:34.822948 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.822902 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466189f1-5bf9-4bb4-8eee-874a7407bc8a-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:08:34.822948 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:34.822911 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbvt7\" (UniqueName: \"kubernetes.io/projected/466189f1-5bf9-4bb4-8eee-874a7407bc8a-kube-api-access-vbvt7\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:08:35.315417 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:35.315377 6389 generic.go:356] "Generic (PLEG): container finished" podID="466189f1-5bf9-4bb4-8eee-874a7407bc8a" containerID="91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2" exitCode=0 Nov 03 15:08:35.315660 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:35.315425 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" Nov 03 15:08:35.315660 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:35.315428 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" event={"ID":"466189f1-5bf9-4bb4-8eee-874a7407bc8a","Type":"ContainerDied","Data":"91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2"} Nov 03 15:08:35.315660 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:35.315566 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5757765976-lklsf" event={"ID":"466189f1-5bf9-4bb4-8eee-874a7407bc8a","Type":"ContainerDied","Data":"8df03a24051560694724064ebb10da10cd50a748b36aba83c60361b8eafcdb9d"} Nov 03 15:08:35.315660 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:35.315597 6389 scope.go:122] "RemoveContainer" containerID="91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2" Nov 03 15:08:35.327886 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:35.327864 6389 scope.go:122] "RemoveContainer" containerID="91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2" Nov 03 15:08:35.328240 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:08:35.328214 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2\": container with ID starting with 91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2 not found: ID does not exist" containerID="91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2" Nov 03 15:08:35.328372 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:35.328246 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2"} err="failed to get container status \"91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2\": rpc error: code = NotFound desc = could not find container \"91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2\": container with ID starting with 91d040a04ccf906c495ccca26155e95b1b80a5a1b7a8377e4ca2249d1b0a07d2 not found: ID does not exist" Nov 03 15:08:35.379786 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:35.379745 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5757765976-lklsf"] Nov 03 15:08:35.393233 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:35.393204 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5757765976-lklsf"] Nov 03 15:08:36.618146 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:08:36.618113 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="466189f1-5bf9-4bb4-8eee-874a7407bc8a" path="/var/lib/kubelet/pods/466189f1-5bf9-4bb4-8eee-874a7407bc8a/volumes" Nov 03 15:08:58.372371 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:08:58.372328 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:09:28.373252 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:09:28.373214 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:09:58.372929 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:09:58.372891 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:10:28.374185 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:10:28.374148 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:10:58.373783 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:10:58.373735 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:11:28.373365 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:11:28.373327 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:11:58.375505 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:11:58.375455 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:12:28.373578 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:12:28.373539 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:12:58.373874 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:12:58.373835 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:13:28.373240 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:13:28.373201 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:13:58.373797 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:13:58.373716 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:14:13.448305 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:13.448263 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs"] Nov 03 15:14:13.448842 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:13.448516 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" containerID="cri-o://aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424" gracePeriod=30 Nov 03 15:14:13.461761 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:13.461728 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28"] Nov 03 15:14:13.461992 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:13.461967 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" containerID="cri-o://9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1" gracePeriod=30 Nov 03 15:14:13.776670 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:13.776557 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z7p28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:14:13.776853 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:13.776658 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:14:18.175715 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.175671 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5btvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 15:14:18.176173 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.175736 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 15:14:18.735978 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.735948 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 15:14:18.754158 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.754135 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 15:14:18.852596 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.852556 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-credential-socket\") pod \"8d9f9066-8666-4172-bf60-4ff8759555bb\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " Nov 03 15:14:18.852813 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.852649 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-podinfo\") pod \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " Nov 03 15:14:18.852813 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.852685 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d9f9066-8666-4172-bf60-4ff8759555bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f9066-8666-4172-bf60-4ff8759555bb-istiod-ca-cert\") pod \"8d9f9066-8666-4172-bf60-4ff8759555bb\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " Nov 03 15:14:18.852813 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.852721 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-data\") pod \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " Nov 03 15:14:18.852813 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.852753 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-certs\") pod \"8d9f9066-8666-4172-bf60-4ff8759555bb\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " Nov 03 15:14:18.852813 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.852809 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-kube-api-access-xw5hb\" (UniqueName: \"kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-kube-api-access-xw5hb\") pod \"8d9f9066-8666-4172-bf60-4ff8759555bb\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " Nov 03 15:14:18.853113 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.852871 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-socket\") pod \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " Nov 03 15:14:18.853113 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.852901 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-credential-socket\") pod \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " Nov 03 15:14:18.853113 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.852923 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-credential-socket" pod "8d9f9066-8666-4172-bf60-4ff8759555bb" (UID: "8d9f9066-8666-4172-bf60-4ff8759555bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.853113 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.852934 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-kube-api-access-n2vsc\" (UniqueName: \"kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-kube-api-access-n2vsc\") pod \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " Nov 03 15:14:18.853113 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.852991 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-socket\") pod \"8d9f9066-8666-4172-bf60-4ff8759555bb\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " Nov 03 15:14:18.853113 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.853064 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-podinfo\") pod \"8d9f9066-8666-4172-bf60-4ff8759555bb\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " Nov 03 15:14:18.853113 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.853091 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-certs\") pod \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " Nov 03 15:14:18.853497 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.853141 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-token\" (UniqueName: \"kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-token\") pod \"8d9f9066-8666-4172-bf60-4ff8759555bb\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " Nov 03 15:14:18.853497 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.853168 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-data\") pod \"8d9f9066-8666-4172-bf60-4ff8759555bb\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " Nov 03 15:14:18.853497 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.853197 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-envoy\") pod \"8d9f9066-8666-4172-bf60-4ff8759555bb\" (UID: \"8d9f9066-8666-4172-bf60-4ff8759555bb\") " Nov 03 15:14:18.853497 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.853228 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istiod-ca-cert\") pod \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " Nov 03 15:14:18.853497 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.853270 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-envoy\") pod \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " Nov 03 15:14:18.853497 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.853325 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-token\" (UniqueName: \"kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-token\") pod \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\" (UID: \"26c9a0b5-fad1-47ea-b811-168742c6fc6e\") " Nov 03 15:14:18.854125 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.853748 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.854125 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.853780 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-socket" pod "26c9a0b5-fad1-47ea-b811-168742c6fc6e" (UID: "26c9a0b5-fad1-47ea-b811-168742c6fc6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.856239 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.854670 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-credential-socket" pod "26c9a0b5-fad1-47ea-b811-168742c6fc6e" (UID: "26c9a0b5-fad1-47ea-b811-168742c6fc6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.856239 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.855116 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9f9066-8666-4172-bf60-4ff8759555bb-istiod-ca-cert" pod "8d9f9066-8666-4172-bf60-4ff8759555bb" (UID: "8d9f9066-8666-4172-bf60-4ff8759555bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:14:18.856239 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.855209 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-certs" pod "8d9f9066-8666-4172-bf60-4ff8759555bb" (UID: "8d9f9066-8666-4172-bf60-4ff8759555bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.856239 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.855409 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-data" pod "26c9a0b5-fad1-47ea-b811-168742c6fc6e" (UID: "26c9a0b5-fad1-47ea-b811-168742c6fc6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.856239 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.855575 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-certs" pod "26c9a0b5-fad1-47ea-b811-168742c6fc6e" (UID: "26c9a0b5-fad1-47ea-b811-168742c6fc6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.856239 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.855772 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-socket" pod "8d9f9066-8666-4172-bf60-4ff8759555bb" (UID: "8d9f9066-8666-4172-bf60-4ff8759555bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.856239 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.855952 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-data" pod "8d9f9066-8666-4172-bf60-4ff8759555bb" (UID: "8d9f9066-8666-4172-bf60-4ff8759555bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.856239 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.856199 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istiod-ca-cert" pod "26c9a0b5-fad1-47ea-b811-168742c6fc6e" (UID: "26c9a0b5-fad1-47ea-b811-168742c6fc6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:14:18.857786 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.857758 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-kube-api-access-n2vsc" pod "26c9a0b5-fad1-47ea-b811-168742c6fc6e" (UID: "26c9a0b5-fad1-47ea-b811-168742c6fc6e"). InnerVolumeSpecName "kube-api-access-n2vsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:14:18.858993 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.858971 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-token" pod "26c9a0b5-fad1-47ea-b811-168742c6fc6e" (UID: "26c9a0b5-fad1-47ea-b811-168742c6fc6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:14:18.859410 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.859387 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-envoy" pod "8d9f9066-8666-4172-bf60-4ff8759555bb" (UID: "8d9f9066-8666-4172-bf60-4ff8759555bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.859502 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.859468 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-envoy" pod "26c9a0b5-fad1-47ea-b811-168742c6fc6e" (UID: "26c9a0b5-fad1-47ea-b811-168742c6fc6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.860272 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.860237 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-token" pod "8d9f9066-8666-4172-bf60-4ff8759555bb" (UID: "8d9f9066-8666-4172-bf60-4ff8759555bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:14:18.861674 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.861601 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-kube-api-access-xw5hb" pod "8d9f9066-8666-4172-bf60-4ff8759555bb" (UID: "8d9f9066-8666-4172-bf60-4ff8759555bb"). InnerVolumeSpecName "kube-api-access-xw5hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:14:18.861991 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.861974 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-podinfo" pod "26c9a0b5-fad1-47ea-b811-168742c6fc6e" (UID: "26c9a0b5-fad1-47ea-b811-168742c6fc6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:14:18.869020 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.868991 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-podinfo" pod "8d9f9066-8666-4172-bf60-4ff8759555bb" (UID: "8d9f9066-8666-4172-bf60-4ff8759555bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:14:18.908476 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.908436 6389 generic.go:356] "Generic (PLEG): container finished" podID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerID="aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424" exitCode=0 Nov 03 15:14:18.908708 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.908651 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" Nov 03 15:14:18.909763 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.909128 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" event={"ID":"26c9a0b5-fad1-47ea-b811-168742c6fc6e","Type":"ContainerDied","Data":"aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424"} Nov 03 15:14:18.909763 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.909203 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs" event={"ID":"26c9a0b5-fad1-47ea-b811-168742c6fc6e","Type":"ContainerDied","Data":"b4cbfa819d672db6e3e99e65651a1198a6d93e54c90fea3c09a30ccfb3a3cfe4"} Nov 03 15:14:18.909763 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.909238 6389 scope.go:122] "RemoveContainer" containerID="aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424" Nov 03 15:14:18.913330 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.913265 6389 generic.go:356] "Generic (PLEG): container finished" podID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerID="9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1" exitCode=0 Nov 03 15:14:18.913330 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.913306 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" event={"ID":"8d9f9066-8666-4172-bf60-4ff8759555bb","Type":"ContainerDied","Data":"9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1"} Nov 03 15:14:18.913330 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.913323 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" Nov 03 15:14:18.913538 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.913364 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28" event={"ID":"8d9f9066-8666-4172-bf60-4ff8759555bb","Type":"ContainerDied","Data":"92e74b2972378dc092b63e75f8d790560283fc1658e9a3769f4b98d9b5611290"} Nov 03 15:14:18.922015 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.921992 6389 scope.go:122] "RemoveContainer" containerID="aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424" Nov 03 15:14:18.922350 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:14:18.922329 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424\": container with ID starting with aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424 not found: ID does not exist" containerID="aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424" Nov 03 15:14:18.922422 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.922358 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424"} err="failed to get container status \"aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424\": rpc error: code = NotFound desc = could not find container \"aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424\": container with ID starting with aa227ea38f37b06f7f0084a477b30a921eddf756d718c836d48a019a2c77e424 not found: ID does not exist" Nov 03 15:14:18.922422 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.922374 6389 scope.go:122] "RemoveContainer" containerID="9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1" Nov 03 15:14:18.932994 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.932974 6389 scope.go:122] "RemoveContainer" containerID="9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1" Nov 03 15:14:18.933336 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:14:18.933315 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1\": container with ID starting with 9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1 not found: ID does not exist" containerID="9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1" Nov 03 15:14:18.933441 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.933343 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1"} err="failed to get container status \"9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1\": rpc error: code = NotFound desc = could not find container \"9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1\": container with ID starting with 9f9d4440c28fe1e4759568c6fe13280d97f7452e687b9799d2fd052baa59ede1 not found: ID does not exist" Nov 03 15:14:18.954219 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954192 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954219 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954216 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f9066-8666-4172-bf60-4ff8759555bb-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954230 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954239 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954247 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw5hb\" (UniqueName: \"kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-kube-api-access-xw5hb\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954256 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954266 6389 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-credential-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954274 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vsc\" (UniqueName: \"kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-kube-api-access-n2vsc\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954283 6389 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-workload-socket\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954296 6389 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-podinfo\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954308 6389 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-workload-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954319 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954326 6389 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-data\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954333 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9f9066-8666-4172-bf60-4ff8759555bb-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954340 6389 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istiod-ca-cert\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954347 6389 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-envoy\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.954424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.954355 6389 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c9a0b5-fad1-47ea-b811-168742c6fc6e-istio-token\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.965070 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.965043 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28"] Nov 03 15:14:18.969072 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:18.969048 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z7p28"] Nov 03 15:14:19.017379 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:19.017303 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs"] Nov 03 15:14:19.021540 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:19.021515 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5btvs"] Nov 03 15:14:20.616932 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:20.616900 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" path="/var/lib/kubelet/pods/26c9a0b5-fad1-47ea-b811-168742c6fc6e/volumes" Nov 03 15:14:20.617342 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:14:20.617328 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" path="/var/lib/kubelet/pods/8d9f9066-8666-4172-bf60-4ff8759555bb/volumes" Nov 03 15:14:28.373520 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:14:28.373421 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:14:58.372993 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:14:58.372955 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:15:28.373431 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:15:28.373384 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:15:58.374395 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:15:58.374340 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:16:28.372762 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:16:28.372722 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:16:28.860755 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.860719 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9"] Nov 03 15:16:28.861291 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.861269 6389 cpu_manager.go:414] "containerMap: removing container" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" Nov 03 15:16:28.861408 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.861293 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" Nov 03 15:16:28.861408 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.861332 6389 cpu_manager.go:414] "containerMap: removing container" podUID="466189f1-5bf9-4bb4-8eee-874a7407bc8a" containerName="maas-api" Nov 03 15:16:28.861408 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.861341 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466189f1-5bf9-4bb4-8eee-874a7407bc8a" containerName="maas-api" Nov 03 15:16:28.861408 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.861361 6389 cpu_manager.go:414] "containerMap: removing container" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" Nov 03 15:16:28.861408 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.861368 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" Nov 03 15:16:28.861602 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.861450 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="466189f1-5bf9-4bb4-8eee-874a7407bc8a" containerName="maas-api" Nov 03 15:16:28.861602 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.861460 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c9a0b5-fad1-47ea-b811-168742c6fc6e" containerName="istio-proxy" Nov 03 15:16:28.861602 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.861470 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d9f9066-8666-4172-bf60-4ff8759555bb" containerName="istio-proxy" Nov 03 15:16:28.861602 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.861481 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d32c7b-9523-4762-8705-a5918ece62c8" containerName="istio-proxy" Nov 03 15:16:28.865029 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.864994 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:28.891600 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.891572 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9"] Nov 03 15:16:28.963009 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.962971 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:28.963210 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.963046 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:28.963210 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.963077 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:28.963210 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.963107 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5k4\" (UniqueName: \"kubernetes.io/projected/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-kube-api-access-gq5k4\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:28.963210 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.963198 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:28.963428 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.963223 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:28.963428 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.963366 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:28.963428 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.963420 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:28.963544 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:28.963478 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.064277 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.064239 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.064277 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.064280 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.064545 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.064305 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.064545 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.064364 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.064545 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.064397 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.064545 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.064433 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5k4\" (UniqueName: \"kubernetes.io/projected/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-kube-api-access-gq5k4\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.064545 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.064464 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.064545 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.064492 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.064545 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.064546 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.064955 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.064836 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.064955 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.064850 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.065066 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.064979 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.065125 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.065070 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.065310 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.065129 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.066951 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.066927 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.067147 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.067128 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.087227 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.087192 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.087349 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.087312 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5k4\" (UniqueName: \"kubernetes.io/projected/fda036b9-bdd2-41a1-b2d5-bddc0f0e396e-kube-api-access-gq5k4\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-dzkt9\" (UID: \"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.179166 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.179135 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:29.495237 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.495204 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9"] Nov 03 15:16:29.497401 ip-10-0-130-191 kubenswrapper[6389]: W1103 15:16:29.497371 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda036b9_bdd2_41a1_b2d5_bddc0f0e396e.slice/crio-dbaa9e09e386e9166930875860b48a090240e0affad2a822daccb525f001396d WatchSource:0}: Error finding container dbaa9e09e386e9166930875860b48a090240e0affad2a822daccb525f001396d: Status 404 returned error can't find the container with id dbaa9e09e386e9166930875860b48a090240e0affad2a822daccb525f001396d Nov 03 15:16:29.499636 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.499589 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 15:16:29.499744 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.499683 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 15:16:29.499744 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.499725 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 15:16:29.508850 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:29.508825 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" event={"ID":"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e","Type":"ContainerStarted","Data":"dbaa9e09e386e9166930875860b48a090240e0affad2a822daccb525f001396d"} Nov 03 15:16:30.514829 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:30.514787 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" event={"ID":"fda036b9-bdd2-41a1-b2d5-bddc0f0e396e","Type":"ContainerStarted","Data":"853390a9ab0cbe805c605934a7d7414a302383dc3d376e1526880a72a59232b5"} Nov 03 15:16:30.593140 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:30.593087 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" podStartSLOduration=2.593073466 podStartE2EDuration="2.593073466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:16:30.579976647 +0000 UTC m=+3866.138613155" watchObservedRunningTime="2026-11-03 15:16:30.593073466 +0000 UTC m=+3866.151709928" Nov 03 15:16:31.180260 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:31.180212 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:31.181723 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:31.181696 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 15:16:31.181855 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:31.181745 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" podUID="fda036b9-bdd2-41a1-b2d5-bddc0f0e396e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 15:16:32.180283 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:32.180241 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 15:16:32.180758 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:32.180299 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" podUID="fda036b9-bdd2-41a1-b2d5-bddc0f0e396e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 15:16:33.179781 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:33.179743 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 15:16:33.179988 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:33.179799 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" podUID="fda036b9-bdd2-41a1-b2d5-bddc0f0e396e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 15:16:34.180329 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:34.180295 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 15:16:34.180794 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:34.180363 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" podUID="fda036b9-bdd2-41a1-b2d5-bddc0f0e396e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 15:16:35.180342 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:35.180301 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 15:16:35.180770 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:35.180358 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" podUID="fda036b9-bdd2-41a1-b2d5-bddc0f0e396e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 15:16:36.179581 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:36.179536 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 15:16:36.179783 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:36.179651 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" podUID="fda036b9-bdd2-41a1-b2d5-bddc0f0e396e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 15:16:37.180290 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:37.180243 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 15:16:37.180737 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:37.180337 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" podUID="fda036b9-bdd2-41a1-b2d5-bddc0f0e396e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 15:16:38.924823 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:38.924790 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:38.925293 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:38.925272 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:38.926265 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:16:38.926243 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" Nov 03 15:16:58.372911 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:16:58.372835 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:17:28.373848 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:17:28.373800 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:17:39.966799 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:39.966746 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:17:39.967458 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:39.966828 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" podUID="fda036b9-bdd2-41a1-b2d5-bddc0f0e396e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:17:58.373900 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:17:58.373858 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:17:59.129895 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.129862 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp"] Nov 03 15:17:59.133838 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.133816 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.166012 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.165983 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp"] Nov 03 15:17:59.242284 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.242250 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.242461 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.242304 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.242461 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.242410 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.242538 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.242474 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.242538 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.242501 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.242624 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.242550 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.242624 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.242581 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdzwn\" (UniqueName: \"kubernetes.io/projected/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-kube-api-access-wdzwn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.242693 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.242624 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.242757 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.242739 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344021 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.343986 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344021 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.344029 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdzwn\" (UniqueName: \"kubernetes.io/projected/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-kube-api-access-wdzwn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344275 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.344056 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344275 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.344117 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344275 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.344151 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344429 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.344270 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344429 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.344335 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344429 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.344398 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344591 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.344435 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344737 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.344713 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344804 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.344751 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344870 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.344842 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.344935 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.344831 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.345060 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.345038 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.346576 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.346557 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.346822 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.346804 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.367817 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.367793 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.368090 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.368071 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdzwn\" (UniqueName: \"kubernetes.io/projected/69abdd4b-9f73-4ae6-b1c8-e5100f15ad80-kube-api-access-wdzwn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lnxjp\" (UID: \"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.450204 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.450170 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:17:59.766867 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.766843 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp"] Nov 03 15:17:59.768691 ip-10-0-130-191 kubenswrapper[6389]: W1103 15:17:59.768660 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69abdd4b_9f73_4ae6_b1c8_e5100f15ad80.slice/crio-8f9727a8784d8eabcac3269f6b352da20db0fe29add08b8495d182e3abc02fa1 WatchSource:0}: Error finding container 8f9727a8784d8eabcac3269f6b352da20db0fe29add08b8495d182e3abc02fa1: Status 404 returned error can't find the container with id 8f9727a8784d8eabcac3269f6b352da20db0fe29add08b8495d182e3abc02fa1 Nov 03 15:17:59.770946 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.770913 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 15:17:59.771028 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.770996 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 15:17:59.771073 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.771025 6389 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 15:17:59.940995 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.940962 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" event={"ID":"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80","Type":"ContainerStarted","Data":"a49fc852243265b6d9181038789979f4639be193e6da833f71777fad3a81a096"} Nov 03 15:17:59.941205 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.941013 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" event={"ID":"69abdd4b-9f73-4ae6-b1c8-e5100f15ad80","Type":"ContainerStarted","Data":"8f9727a8784d8eabcac3269f6b352da20db0fe29add08b8495d182e3abc02fa1"} Nov 03 15:17:59.987531 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:17:59.987421 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" podStartSLOduration=0.987406523 podStartE2EDuration="987.406523ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:59.986180637 +0000 UTC m=+3955.544817121" watchObservedRunningTime="2026-11-03 15:17:59.987406523 +0000 UTC m=+3955.546043009" Nov 03 15:18:00.450565 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:00.450523 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:18:00.452044 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:00.452020 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:18:00.452100 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:00.452071 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" podUID="69abdd4b-9f73-4ae6-b1c8-e5100f15ad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:18:01.451126 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:01.451084 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:18:01.451551 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:01.451147 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" podUID="69abdd4b-9f73-4ae6-b1c8-e5100f15ad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:18:02.450699 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:02.450660 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:18:02.450894 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:02.450721 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" podUID="69abdd4b-9f73-4ae6-b1c8-e5100f15ad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:18:03.451214 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:03.451167 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:18:03.451680 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:03.451246 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" podUID="69abdd4b-9f73-4ae6-b1c8-e5100f15ad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:18:04.450868 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:04.450834 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:18:04.451136 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:04.450901 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" podUID="69abdd4b-9f73-4ae6-b1c8-e5100f15ad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:18:05.451346 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:05.451311 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:18:05.451801 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:05.451384 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" podUID="69abdd4b-9f73-4ae6-b1c8-e5100f15ad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:18:06.450982 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:06.450945 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:18:06.451174 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:06.451012 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" podUID="69abdd4b-9f73-4ae6-b1c8-e5100f15ad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:18:07.451316 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:07.451277 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 15:18:07.451760 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:07.451342 6389 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" podUID="69abdd4b-9f73-4ae6-b1c8-e5100f15ad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 15:18:08.993565 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:08.993531 6389 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:18:08.994079 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:08.994058 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:18:08.994860 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:18:08.994841 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" Nov 03 15:18:28.374456 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:18:28.374418 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:18:58.373273 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:18:58.373229 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:19:12.366237 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.366202 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c"] Nov 03 15:19:12.369515 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.369499 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:12.388176 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.388152 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7b218f2f-serving-cert\"" Nov 03 15:19:12.401172 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.401141 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c"] Nov 03 15:19:12.467275 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.467237 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-metrics-certs\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:12.467275 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.467272 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-maas-api-tls\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:12.467537 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.467296 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b366a1f9-f0ec-4988-8568-fe081263b533-openshift-service-ca\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:12.467537 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.467357 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676rg\" (UniqueName: \"kubernetes.io/projected/b366a1f9-f0ec-4988-8568-fe081263b533-kube-api-access-676rg\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:12.568084 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.568050 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-metrics-certs\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:12.568084 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.568087 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-maas-api-tls\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:12.568307 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.568107 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b366a1f9-f0ec-4988-8568-fe081263b533-openshift-service-ca\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:12.568307 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.568122 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676rg\" (UniqueName: \"kubernetes.io/projected/b366a1f9-f0ec-4988-8568-fe081263b533-kube-api-access-676rg\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:12.568307 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:19:12.568224 6389 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-serving-cert: secret "maas-api-e2e-mt-7b218f2f-serving-cert" not found Nov 03 15:19:12.568307 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:19:12.568299 6389 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-maas-api-tls podName:b366a1f9-f0ec-4988-8568-fe081263b533 nodeName:}" failed. No retries permitted until 2026-11-03 15:19:13.06827952 +0000 UTC m=+4028.626915981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-maas-api-tls") pod "maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" (UID: "b366a1f9-f0ec-4988-8568-fe081263b533") : secret "maas-api-e2e-mt-7b218f2f-serving-cert" not found Nov 03 15:19:12.568858 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.568840 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b366a1f9-f0ec-4988-8568-fe081263b533-openshift-service-ca\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:12.570691 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.570672 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-metrics-certs\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:12.590869 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.590846 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-676rg\" (UniqueName: \"kubernetes.io/projected/b366a1f9-f0ec-4988-8568-fe081263b533-kube-api-access-676rg\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:12.674841 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.674808 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj"] Nov 03 15:19:12.678419 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.678401 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" Nov 03 15:19:12.698628 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.698588 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7b218f2f-dockercfg-lh4bf\"" Nov 03 15:19:12.698819 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.698598 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7b218f2f\"" Nov 03 15:19:12.699504 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.699479 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj"] Nov 03 15:19:12.771007 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.770970 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hxjn\" (UniqueName: \"kubernetes.io/projected/889a18e2-55ac-4e69-8c94-fbff0331af6e-kube-api-access-7hxjn\") pod \"payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj\" (UID: \"889a18e2-55ac-4e69-8c94-fbff0331af6e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" Nov 03 15:19:12.771185 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.771074 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889a18e2-55ac-4e69-8c94-fbff0331af6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj\" (UID: \"889a18e2-55ac-4e69-8c94-fbff0331af6e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" Nov 03 15:19:12.779596 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.779549 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv"] Nov 03 15:19:12.783879 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.783855 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" Nov 03 15:19:12.803774 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.803739 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv"] Nov 03 15:19:12.871883 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.871851 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hxjn\" (UniqueName: \"kubernetes.io/projected/889a18e2-55ac-4e69-8c94-fbff0331af6e-kube-api-access-7hxjn\") pod \"payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj\" (UID: \"889a18e2-55ac-4e69-8c94-fbff0331af6e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" Nov 03 15:19:12.872072 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.871913 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889a18e2-55ac-4e69-8c94-fbff0331af6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj\" (UID: \"889a18e2-55ac-4e69-8c94-fbff0331af6e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" Nov 03 15:19:12.872583 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.872565 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889a18e2-55ac-4e69-8c94-fbff0331af6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj\" (UID: \"889a18e2-55ac-4e69-8c94-fbff0331af6e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" Nov 03 15:19:12.895150 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.895124 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hxjn\" (UniqueName: \"kubernetes.io/projected/889a18e2-55ac-4e69-8c94-fbff0331af6e-kube-api-access-7hxjn\") pod \"payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj\" (UID: \"889a18e2-55ac-4e69-8c94-fbff0331af6e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" Nov 03 15:19:12.973421 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.973325 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frcc\" (UniqueName: \"kubernetes.io/projected/47bae315-cfbd-4703-b48e-a8e40c9ee033-kube-api-access-4frcc\") pod \"payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv\" (UID: \"47bae315-cfbd-4703-b48e-a8e40c9ee033\") " pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" Nov 03 15:19:12.973421 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.973369 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bae315-cfbd-4703-b48e-a8e40c9ee033-plugins-config-volume\") pod \"payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv\" (UID: \"47bae315-cfbd-4703-b48e-a8e40c9ee033\") " pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" Nov 03 15:19:12.991850 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:12.991823 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" Nov 03 15:19:13.074132 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:13.074100 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-maas-api-tls\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:13.074289 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:13.074162 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frcc\" (UniqueName: \"kubernetes.io/projected/47bae315-cfbd-4703-b48e-a8e40c9ee033-kube-api-access-4frcc\") pod \"payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv\" (UID: \"47bae315-cfbd-4703-b48e-a8e40c9ee033\") " pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" Nov 03 15:19:13.074289 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:13.074192 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bae315-cfbd-4703-b48e-a8e40c9ee033-plugins-config-volume\") pod \"payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv\" (UID: \"47bae315-cfbd-4703-b48e-a8e40c9ee033\") " pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" Nov 03 15:19:13.074923 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:13.074858 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bae315-cfbd-4703-b48e-a8e40c9ee033-plugins-config-volume\") pod \"payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv\" (UID: \"47bae315-cfbd-4703-b48e-a8e40c9ee033\") " pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" Nov 03 15:19:13.076791 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:13.076767 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-maas-api-tls\") pod \"maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:13.097602 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:13.097570 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frcc\" (UniqueName: \"kubernetes.io/projected/47bae315-cfbd-4703-b48e-a8e40c9ee033-kube-api-access-4frcc\") pod \"payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv\" (UID: \"47bae315-cfbd-4703-b48e-a8e40c9ee033\") " pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" Nov 03 15:19:13.100451 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:13.100430 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" Nov 03 15:19:13.283081 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:13.283001 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:13.301851 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:13.301830 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj"] Nov 03 15:19:13.303937 ip-10-0-130-191 kubenswrapper[6389]: W1103 15:19:13.303910 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889a18e2_55ac_4e69_8c94_fbff0331af6e.slice/crio-067b7e4e32bfd65dc6a3f0b0ecad4c109875b9c8d9e4a6feb2c58364bc5c8361 WatchSource:0}: Error finding container 067b7e4e32bfd65dc6a3f0b0ecad4c109875b9c8d9e4a6feb2c58364bc5c8361: Status 404 returned error can't find the container with id 067b7e4e32bfd65dc6a3f0b0ecad4c109875b9c8d9e4a6feb2c58364bc5c8361 Nov 03 15:19:13.414184 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:13.414161 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv"] Nov 03 15:19:13.415504 ip-10-0-130-191 kubenswrapper[6389]: W1103 15:19:13.415479 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bae315_cfbd_4703_b48e_a8e40c9ee033.slice/crio-743de59a8b9862d19d152974665fa2402d9528182d8c376e1027ccf343f5498f WatchSource:0}: Error finding container 743de59a8b9862d19d152974665fa2402d9528182d8c376e1027ccf343f5498f: Status 404 returned error can't find the container with id 743de59a8b9862d19d152974665fa2402d9528182d8c376e1027ccf343f5498f Nov 03 15:19:13.588182 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:13.588131 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c"] Nov 03 15:19:13.590682 ip-10-0-130-191 kubenswrapper[6389]: W1103 15:19:13.590651 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb366a1f9_f0ec_4988_8568_fe081263b533.slice/crio-460d6c9f9e46b705c91528a51571c2de08baedbf6c23252309ac4029d2b2c7ff WatchSource:0}: Error finding container 460d6c9f9e46b705c91528a51571c2de08baedbf6c23252309ac4029d2b2c7ff: Status 404 returned error can't find the container with id 460d6c9f9e46b705c91528a51571c2de08baedbf6c23252309ac4029d2b2c7ff Nov 03 15:19:13.592136 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:13.592112 6389 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:19:14.280847 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:14.280811 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" event={"ID":"b366a1f9-f0ec-4988-8568-fe081263b533","Type":"ContainerStarted","Data":"ed06b9941a4652efbee6eeca19a12b4e6ddb47cca07fef3cca770623cf0d31d7"} Nov 03 15:19:14.281080 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:14.280869 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" event={"ID":"b366a1f9-f0ec-4988-8568-fe081263b533","Type":"ContainerStarted","Data":"460d6c9f9e46b705c91528a51571c2de08baedbf6c23252309ac4029d2b2c7ff"} Nov 03 15:19:14.281080 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:14.280910 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:14.282537 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:14.282512 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" event={"ID":"47bae315-cfbd-4703-b48e-a8e40c9ee033","Type":"ContainerStarted","Data":"b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e"} Nov 03 15:19:14.282694 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:14.282558 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" Nov 03 15:19:14.282694 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:14.282575 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" event={"ID":"47bae315-cfbd-4703-b48e-a8e40c9ee033","Type":"ContainerStarted","Data":"743de59a8b9862d19d152974665fa2402d9528182d8c376e1027ccf343f5498f"} Nov 03 15:19:14.284062 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:14.284041 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" event={"ID":"889a18e2-55ac-4e69-8c94-fbff0331af6e","Type":"ContainerStarted","Data":"cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081"} Nov 03 15:19:14.284176 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:14.284086 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" event={"ID":"889a18e2-55ac-4e69-8c94-fbff0331af6e","Type":"ContainerStarted","Data":"067b7e4e32bfd65dc6a3f0b0ecad4c109875b9c8d9e4a6feb2c58364bc5c8361"} Nov 03 15:19:14.284176 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:14.284135 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" Nov 03 15:19:14.325654 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:14.325519 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" podStartSLOduration=1.8682141730000001 podStartE2EDuration="2.325496036s" totalImagesPullingTime="457.281863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:14.324652848 +0000 UTC m=+4029.883289331" watchObservedRunningTime="2026-11-03 15:19:14.325496036 +0000 UTC m=+4029.884132519" Nov 03 15:19:14.367563 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:14.367504 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" podStartSLOduration=2.367486778 podStartE2EDuration="2.367486778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:14.366289164 +0000 UTC m=+4029.924925648" watchObservedRunningTime="2026-11-03 15:19:14.367486778 +0000 UTC m=+4029.926123298" Nov 03 15:19:14.409130 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:14.409077 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" podStartSLOduration=2.409063495 podStartE2EDuration="2.409063495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:14.408509235 +0000 UTC m=+4029.967145719" watchObservedRunningTime="2026-11-03 15:19:14.409063495 +0000 UTC m=+4029.967699985" Nov 03 15:19:20.293983 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:20.293950 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:19:25.291302 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:25.291274 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" Nov 03 15:19:25.291816 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:19:25.291352 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" Nov 03 15:19:28.373148 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:19:28.373112 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:19:58.373839 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:19:58.373750 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:20:28.373833 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:20:28.373794 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:20:58.373561 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:20:58.373522 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:21:06.083743 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:06.083708 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv"] Nov 03 15:21:06.129418 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:06.129383 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj"] Nov 03 15:21:09.629022 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.628985 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c"] Nov 03 15:21:09.629866 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.629819 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" podUID="b366a1f9-f0ec-4988-8568-fe081263b533" containerName="maas-api" containerID="cri-o://ed06b9941a4652efbee6eeca19a12b4e6ddb47cca07fef3cca770623cf0d31d7" gracePeriod=30 Nov 03 15:21:09.860195 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.860166 6389 generic.go:356] "Generic (PLEG): container finished" podID="b366a1f9-f0ec-4988-8568-fe081263b533" containerID="ed06b9941a4652efbee6eeca19a12b4e6ddb47cca07fef3cca770623cf0d31d7" exitCode=0 Nov 03 15:21:09.860327 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.860239 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" event={"ID":"b366a1f9-f0ec-4988-8568-fe081263b533","Type":"ContainerDied","Data":"ed06b9941a4652efbee6eeca19a12b4e6ddb47cca07fef3cca770623cf0d31d7"} Nov 03 15:21:09.860327 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.860291 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" event={"ID":"b366a1f9-f0ec-4988-8568-fe081263b533","Type":"ContainerDied","Data":"460d6c9f9e46b705c91528a51571c2de08baedbf6c23252309ac4029d2b2c7ff"} Nov 03 15:21:09.860327 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.860308 6389 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="460d6c9f9e46b705c91528a51571c2de08baedbf6c23252309ac4029d2b2c7ff" Nov 03 15:21:09.871256 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.871234 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:21:09.960347 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.960314 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-maas-api-tls\") pod \"b366a1f9-f0ec-4988-8568-fe081263b533\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " Nov 03 15:21:09.960546 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.960368 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b366a1f9-f0ec-4988-8568-fe081263b533-kube-api-access-676rg\" (UniqueName: \"kubernetes.io/projected/b366a1f9-f0ec-4988-8568-fe081263b533-kube-api-access-676rg\") pod \"b366a1f9-f0ec-4988-8568-fe081263b533\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " Nov 03 15:21:09.960546 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.960405 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b366a1f9-f0ec-4988-8568-fe081263b533-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b366a1f9-f0ec-4988-8568-fe081263b533-openshift-service-ca\") pod \"b366a1f9-f0ec-4988-8568-fe081263b533\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " Nov 03 15:21:09.960546 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.960441 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-metrics-certs\") pod \"b366a1f9-f0ec-4988-8568-fe081263b533\" (UID: \"b366a1f9-f0ec-4988-8568-fe081263b533\") " Nov 03 15:21:09.960895 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.960867 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b366a1f9-f0ec-4988-8568-fe081263b533-openshift-service-ca" pod "b366a1f9-f0ec-4988-8568-fe081263b533" (UID: "b366a1f9-f0ec-4988-8568-fe081263b533"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:21:09.962829 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.962797 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-maas-api-tls" pod "b366a1f9-f0ec-4988-8568-fe081263b533" (UID: "b366a1f9-f0ec-4988-8568-fe081263b533"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:21:09.962932 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.962829 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b366a1f9-f0ec-4988-8568-fe081263b533-kube-api-access-676rg" pod "b366a1f9-f0ec-4988-8568-fe081263b533" (UID: "b366a1f9-f0ec-4988-8568-fe081263b533"). InnerVolumeSpecName "kube-api-access-676rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:21:09.962974 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:09.962935 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-metrics-certs" pod "b366a1f9-f0ec-4988-8568-fe081263b533" (UID: "b366a1f9-f0ec-4988-8568-fe081263b533"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:21:10.073298 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:10.073249 6389 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-maas-api-tls\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:21:10.073298 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:10.073290 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-676rg\" (UniqueName: \"kubernetes.io/projected/b366a1f9-f0ec-4988-8568-fe081263b533-kube-api-access-676rg\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:21:10.073567 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:10.073306 6389 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b366a1f9-f0ec-4988-8568-fe081263b533-openshift-service-ca\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:21:10.073567 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:10.073320 6389 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b366a1f9-f0ec-4988-8568-fe081263b533-metrics-certs\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:21:10.865198 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:10.865161 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c" Nov 03 15:21:10.931263 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:10.931227 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c"] Nov 03 15:21:10.943792 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:10.943761 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b218f2f-f6f47f788-cxs4c"] Nov 03 15:21:11.084066 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.084024 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" podUID="47bae315-cfbd-4703-b48e-a8e40c9ee033" containerName="payload-processing" containerID="cri-o://b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e" gracePeriod=25 Nov 03 15:21:11.130318 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.130278 6389 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" podUID="889a18e2-55ac-4e69-8c94-fbff0331af6e" containerName="payload-pre-processing" containerID="cri-o://cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081" gracePeriod=25 Nov 03 15:21:11.382645 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.382618 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" Nov 03 15:21:11.406577 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.406556 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" Nov 03 15:21:11.487834 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.487748 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47bae315-cfbd-4703-b48e-a8e40c9ee033-kube-api-access-4frcc\" (UniqueName: \"kubernetes.io/projected/47bae315-cfbd-4703-b48e-a8e40c9ee033-kube-api-access-4frcc\") pod \"47bae315-cfbd-4703-b48e-a8e40c9ee033\" (UID: \"47bae315-cfbd-4703-b48e-a8e40c9ee033\") " Nov 03 15:21:11.487998 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.487883 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47bae315-cfbd-4703-b48e-a8e40c9ee033-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bae315-cfbd-4703-b48e-a8e40c9ee033-plugins-config-volume\") pod \"47bae315-cfbd-4703-b48e-a8e40c9ee033\" (UID: \"47bae315-cfbd-4703-b48e-a8e40c9ee033\") " Nov 03 15:21:11.488333 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.488295 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bae315-cfbd-4703-b48e-a8e40c9ee033-plugins-config-volume" pod "47bae315-cfbd-4703-b48e-a8e40c9ee033" (UID: "47bae315-cfbd-4703-b48e-a8e40c9ee033"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:21:11.490058 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.490038 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47bae315-cfbd-4703-b48e-a8e40c9ee033-kube-api-access-4frcc" pod "47bae315-cfbd-4703-b48e-a8e40c9ee033" (UID: "47bae315-cfbd-4703-b48e-a8e40c9ee033"). InnerVolumeSpecName "kube-api-access-4frcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:21:11.588655 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.588599 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/889a18e2-55ac-4e69-8c94-fbff0331af6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889a18e2-55ac-4e69-8c94-fbff0331af6e-plugins-config-volume\") pod \"889a18e2-55ac-4e69-8c94-fbff0331af6e\" (UID: \"889a18e2-55ac-4e69-8c94-fbff0331af6e\") " Nov 03 15:21:11.588872 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.588773 6389 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889a18e2-55ac-4e69-8c94-fbff0331af6e-kube-api-access-7hxjn\" (UniqueName: \"kubernetes.io/projected/889a18e2-55ac-4e69-8c94-fbff0331af6e-kube-api-access-7hxjn\") pod \"889a18e2-55ac-4e69-8c94-fbff0331af6e\" (UID: \"889a18e2-55ac-4e69-8c94-fbff0331af6e\") " Nov 03 15:21:11.588980 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.588956 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889a18e2-55ac-4e69-8c94-fbff0331af6e-plugins-config-volume" pod "889a18e2-55ac-4e69-8c94-fbff0331af6e" (UID: "889a18e2-55ac-4e69-8c94-fbff0331af6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:21:11.589090 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.589076 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889a18e2-55ac-4e69-8c94-fbff0331af6e-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:21:11.589131 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.589093 6389 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bae315-cfbd-4703-b48e-a8e40c9ee033-plugins-config-volume\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:21:11.589131 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.589103 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4frcc\" (UniqueName: \"kubernetes.io/projected/47bae315-cfbd-4703-b48e-a8e40c9ee033-kube-api-access-4frcc\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:21:11.590925 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.590905 6389 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889a18e2-55ac-4e69-8c94-fbff0331af6e-kube-api-access-7hxjn" pod "889a18e2-55ac-4e69-8c94-fbff0331af6e" (UID: "889a18e2-55ac-4e69-8c94-fbff0331af6e"). InnerVolumeSpecName "kube-api-access-7hxjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:21:11.689981 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.689951 6389 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hxjn\" (UniqueName: \"kubernetes.io/projected/889a18e2-55ac-4e69-8c94-fbff0331af6e-kube-api-access-7hxjn\") on node \"ip-10-0-130-191.ec2.internal\" DevicePath \"\"" Nov 03 15:21:11.871397 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.871315 6389 generic.go:356] "Generic (PLEG): container finished" podID="47bae315-cfbd-4703-b48e-a8e40c9ee033" containerID="b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e" exitCode=0 Nov 03 15:21:11.871397 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.871361 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" Nov 03 15:21:11.871961 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.871407 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" event={"ID":"47bae315-cfbd-4703-b48e-a8e40c9ee033","Type":"ContainerDied","Data":"b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e"} Nov 03 15:21:11.871961 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.871458 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv" event={"ID":"47bae315-cfbd-4703-b48e-a8e40c9ee033","Type":"ContainerDied","Data":"743de59a8b9862d19d152974665fa2402d9528182d8c376e1027ccf343f5498f"} Nov 03 15:21:11.871961 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.871487 6389 scope.go:122] "RemoveContainer" containerID="b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e" Nov 03 15:21:11.872736 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.872716 6389 generic.go:356] "Generic (PLEG): container finished" podID="889a18e2-55ac-4e69-8c94-fbff0331af6e" containerID="cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081" exitCode=0 Nov 03 15:21:11.872839 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.872752 6389 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" Nov 03 15:21:11.872839 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.872813 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" event={"ID":"889a18e2-55ac-4e69-8c94-fbff0331af6e","Type":"ContainerDied","Data":"cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081"} Nov 03 15:21:11.872920 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.872864 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj" event={"ID":"889a18e2-55ac-4e69-8c94-fbff0331af6e","Type":"ContainerDied","Data":"067b7e4e32bfd65dc6a3f0b0ecad4c109875b9c8d9e4a6feb2c58364bc5c8361"} Nov 03 15:21:11.880699 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.880676 6389 scope.go:122] "RemoveContainer" containerID="b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e" Nov 03 15:21:11.881136 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:21:11.880995 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e\": container with ID starting with b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e not found: ID does not exist" containerID="b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e" Nov 03 15:21:11.881136 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.881039 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e"} err="failed to get container status \"b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e\": rpc error: code = NotFound desc = could not find container \"b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e\": container with ID starting with b1f98a1acc5408afa1001652097d0c4ed6b4593f02858edd9da9353c4036690e not found: ID does not exist" Nov 03 15:21:11.881136 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.881060 6389 scope.go:122] "RemoveContainer" containerID="cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081" Nov 03 15:21:11.889655 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.889639 6389 scope.go:122] "RemoveContainer" containerID="cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081" Nov 03 15:21:11.889928 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:21:11.889909 6389 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081\": container with ID starting with cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081 not found: ID does not exist" containerID="cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081" Nov 03 15:21:11.889973 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.889936 6389 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081"} err="failed to get container status \"cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081\": rpc error: code = NotFound desc = could not find container \"cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081\": container with ID starting with cf6bab823aab53282799f8a7fa56b99fa5e64d9bfc24b4314d82398915263081 not found: ID does not exist" Nov 03 15:21:11.935432 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.935402 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj"] Nov 03 15:21:11.939056 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.939032 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b218f2f-66d74bf9b7-xtdpj"] Nov 03 15:21:11.985311 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.985279 6389 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv"] Nov 03 15:21:11.992810 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:11.992778 6389 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b218f2f-8485696fcf-t5bhv"] Nov 03 15:21:12.616428 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:12.616396 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47bae315-cfbd-4703-b48e-a8e40c9ee033" path="/var/lib/kubelet/pods/47bae315-cfbd-4703-b48e-a8e40c9ee033/volumes" Nov 03 15:21:12.616781 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:12.616768 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="889a18e2-55ac-4e69-8c94-fbff0331af6e" path="/var/lib/kubelet/pods/889a18e2-55ac-4e69-8c94-fbff0331af6e/volumes" Nov 03 15:21:12.617131 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:12.617119 6389 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b366a1f9-f0ec-4988-8568-fe081263b533" path="/var/lib/kubelet/pods/b366a1f9-f0ec-4988-8568-fe081263b533/volumes" Nov 03 15:21:28.373973 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:21:28.373932 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:21:55.035845 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:55.035799 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:21:55.036455 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:55.035890 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" podUID="69abdd4b-9f73-4ae6-b1c8-e5100f15ad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:21:58.378904 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:21:58.378861 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:21:59.953737 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:59.953695 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:21:59.954347 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:21:59.954313 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:22:28.374072 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:22:28.374037 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:22:58.373378 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:22:58.373291 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:23:09.996490 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:23:09.995705 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:23:09.996490 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:23:09.995771 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-dzkt9" podUID="fda036b9-bdd2-41a1-b2d5-bddc0f0e396e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:23:10.047780 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:23:10.047738 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:23:10.048142 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:23:10.048110 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" podUID="69abdd4b-9f73-4ae6-b1c8-e5100f15ad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:23:28.373081 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:23:28.373037 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:23:58.412650 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:23:58.412595 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:23:59.945752 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:23:59.945708 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:23:59.946986 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:23:59.946951 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7jwxs" podUID="e5733a68-00fa-4304-87d1-e10764bf1d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:24:28.373001 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:24:28.372964 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:24:40.041633 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:24:40.040754 6389 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:24:40.041633 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:24:40.040857 6389 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lnxjp" podUID="69abdd4b-9f73-4ae6-b1c8-e5100f15ad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:24:58.373829 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:24:58.373793 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:25:28.375645 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:25:28.375574 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:25:58.373327 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:25:58.373238 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:26:07.800072 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:07.800039 6389 scope.go:122] "RemoveContainer" containerID="ed06b9941a4652efbee6eeca19a12b4e6ddb47cca07fef3cca770623cf0d31d7" Nov 03 15:26:28.373260 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:26:28.373218 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:26:33.075098 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:33.075070 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5f464594c6-562pp_428b2a8c-5dc4-4b29-a535-35b8b20cf721/manager/0.log" Nov 03 15:26:33.548523 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:33.548491 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qdflj_f884e514-a7a4-459f-aba9-62f7ce76671a/manager/0.log" Nov 03 15:26:34.067285 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:34.067231 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dfdc587c5-vbgqs_fc2d0b86-48fd-48f6-89fa-150de7e46ca9/manager/0.log" Nov 03 15:26:35.843042 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:35.843010 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84b5dd6748-qqjc7_3a5591b5-8964-4d8b-b94d-934ff325b6aa/maas-api/0.log" Nov 03 15:26:39.307148 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:39.307115 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qpxtd_5e2b48a3-4d88-4851-86f3-cd8bcc0292c0/postgres/0.log" Nov 03 15:26:41.771793 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:41.771756 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-k9lp2_bf69a67f-797a-4489-a929-789d47efc69c/kuadrant-console-plugin/0.log" Nov 03 15:26:43.618247 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:43.618201 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-m6wvl_ea0ab114-50e5-4549-8af4-f361d3f23df7/discovery/0.log" Nov 03 15:26:44.007574 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:44.007522 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76667dfcdc-rj9l9_502f2df2-d66f-4023-acb6-6ef16ac7f5a8/kube-auth-proxy/0.log" Nov 03 15:26:44.593348 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:44.593317 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-7jwxs_e5733a68-00fa-4304-87d1-e10764bf1d6c/istio-proxy/0.log" Nov 03 15:26:45.186816 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:45.186781 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-dzkt9_fda036b9-bdd2-41a1-b2d5-bddc0f0e396e/istio-proxy/0.log" Nov 03 15:26:45.381872 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:45.381836 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-lnxjp_69abdd4b-9f73-4ae6-b1c8-e5100f15ad80/istio-proxy/0.log" Nov 03 15:26:46.323857 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:46.323825 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76bfcf8f7d-tzw9j_8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69/payload-pre-processing/0.log" Nov 03 15:26:47.663037 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:47.663010 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5qfl4_42f57c84-c009-4eab-8029-633dc6e21aa2/main/0.log" Nov 03 15:26:47.848394 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:47.848366 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-d77tr_d87e0fff-43c1-4030-b7bc-9c16a38e6872/main/0.log" Nov 03 15:26:48.398941 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:48.398905 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8b6c6n_35993ab8-aaa0-41cd-b4cf-5f0e85fb11d2/main/0.log" Nov 03 15:26:48.596208 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:48.596175 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-c242s_d3d2a960-6069-4a49-a738-e07f9178d7a0/main/0.log" Nov 03 15:26:48.789583 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:48.789552 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-vvxk9_5f9d65ca-012a-4efa-a43b-2a11844f241e/main/0.log" Nov 03 15:26:58.373782 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:26:58.373749 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:26:58.978908 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:58.978871 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rfl4d/must-gather-hmq69"] Nov 03 15:26:58.979277 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:58.979261 6389 cpu_manager.go:414] "containerMap: removing container" podUID="b366a1f9-f0ec-4988-8568-fe081263b533" containerName="maas-api" Nov 03 15:26:58.979333 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:58.979278 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b366a1f9-f0ec-4988-8568-fe081263b533" containerName="maas-api" Nov 03 15:26:58.979333 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:58.979299 6389 cpu_manager.go:414] "containerMap: removing container" podUID="47bae315-cfbd-4703-b48e-a8e40c9ee033" containerName="payload-processing" Nov 03 15:26:58.979333 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:58.979305 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47bae315-cfbd-4703-b48e-a8e40c9ee033" containerName="payload-processing" Nov 03 15:26:58.979333 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:58.979325 6389 cpu_manager.go:414] "containerMap: removing container" podUID="889a18e2-55ac-4e69-8c94-fbff0331af6e" containerName="payload-pre-processing" Nov 03 15:26:58.979333 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:58.979330 6389 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="889a18e2-55ac-4e69-8c94-fbff0331af6e" containerName="payload-pre-processing" Nov 03 15:26:58.979519 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:58.979400 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="889a18e2-55ac-4e69-8c94-fbff0331af6e" containerName="payload-pre-processing" Nov 03 15:26:58.979519 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:58.979408 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="47bae315-cfbd-4703-b48e-a8e40c9ee033" containerName="payload-processing" Nov 03 15:26:58.979519 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:58.979417 6389 memory_manager.go:365] "RemoveStaleState removing state" podUID="b366a1f9-f0ec-4988-8568-fe081263b533" containerName="maas-api" Nov 03 15:26:58.982681 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:58.982661 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rfl4d/must-gather-hmq69" Nov 03 15:26:59.013910 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.013886 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rfl4d\"/\"kube-root-ca.crt\"" Nov 03 15:26:59.014095 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.013956 6389 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rfl4d\"/\"openshift-service-ca.crt\"" Nov 03 15:26:59.014095 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.013967 6389 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rfl4d\"/\"default-dockercfg-nh78f\"" Nov 03 15:26:59.015693 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.015667 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rfl4d/must-gather-hmq69"] Nov 03 15:26:59.056504 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.056467 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wfsh\" (UniqueName: \"kubernetes.io/projected/99df25ad-0e70-4fb3-a603-faf083975110-kube-api-access-6wfsh\") pod \"must-gather-hmq69\" (UID: \"99df25ad-0e70-4fb3-a603-faf083975110\") " pod="openshift-must-gather-rfl4d/must-gather-hmq69" Nov 03 15:26:59.056725 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.056602 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99df25ad-0e70-4fb3-a603-faf083975110-must-gather-output\") pod \"must-gather-hmq69\" (UID: \"99df25ad-0e70-4fb3-a603-faf083975110\") " pod="openshift-must-gather-rfl4d/must-gather-hmq69" Nov 03 15:26:59.157451 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.157416 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99df25ad-0e70-4fb3-a603-faf083975110-must-gather-output\") pod \"must-gather-hmq69\" (UID: \"99df25ad-0e70-4fb3-a603-faf083975110\") " pod="openshift-must-gather-rfl4d/must-gather-hmq69" Nov 03 15:26:59.157658 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.157469 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wfsh\" (UniqueName: \"kubernetes.io/projected/99df25ad-0e70-4fb3-a603-faf083975110-kube-api-access-6wfsh\") pod \"must-gather-hmq69\" (UID: \"99df25ad-0e70-4fb3-a603-faf083975110\") " pod="openshift-must-gather-rfl4d/must-gather-hmq69" Nov 03 15:26:59.157837 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.157814 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99df25ad-0e70-4fb3-a603-faf083975110-must-gather-output\") pod \"must-gather-hmq69\" (UID: \"99df25ad-0e70-4fb3-a603-faf083975110\") " pod="openshift-must-gather-rfl4d/must-gather-hmq69" Nov 03 15:26:59.180414 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.180378 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wfsh\" (UniqueName: \"kubernetes.io/projected/99df25ad-0e70-4fb3-a603-faf083975110-kube-api-access-6wfsh\") pod \"must-gather-hmq69\" (UID: \"99df25ad-0e70-4fb3-a603-faf083975110\") " pod="openshift-must-gather-rfl4d/must-gather-hmq69" Nov 03 15:26:59.297430 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.297345 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rfl4d/must-gather-hmq69" Nov 03 15:26:59.613868 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.613841 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rfl4d/must-gather-hmq69"] Nov 03 15:26:59.616006 ip-10-0-130-191 kubenswrapper[6389]: W1103 15:26:59.615979 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99df25ad_0e70_4fb3_a603_faf083975110.slice/crio-5b5ec2ba7679b45f08e2eeccb8f730e4a4852a17d8918b2235ed816c2df9f7e2 WatchSource:0}: Error finding container 5b5ec2ba7679b45f08e2eeccb8f730e4a4852a17d8918b2235ed816c2df9f7e2: Status 404 returned error can't find the container with id 5b5ec2ba7679b45f08e2eeccb8f730e4a4852a17d8918b2235ed816c2df9f7e2 Nov 03 15:26:59.618047 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:26:59.618029 6389 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:27:00.023346 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:00.023315 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rfl4d/must-gather-hmq69" event={"ID":"99df25ad-0e70-4fb3-a603-faf083975110","Type":"ContainerStarted","Data":"5b5ec2ba7679b45f08e2eeccb8f730e4a4852a17d8918b2235ed816c2df9f7e2"} Nov 03 15:27:01.038640 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:01.038577 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rfl4d/must-gather-hmq69" event={"ID":"99df25ad-0e70-4fb3-a603-faf083975110","Type":"ContainerStarted","Data":"91157457340f63b5599f225819ad4f62876629dbda5ecbb521ebd16542d44138"} Nov 03 15:27:01.039206 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:01.038663 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rfl4d/must-gather-hmq69" event={"ID":"99df25ad-0e70-4fb3-a603-faf083975110","Type":"ContainerStarted","Data":"900326f88f665341e6aa09b919d6c9fd9562da07781b1c850d155068f4b4adfa"} Nov 03 15:27:01.086978 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:01.086927 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rfl4d/must-gather-hmq69" podStartSLOduration=2.27599831 podStartE2EDuration="3.086912931s" totalImagesPullingTime="810.914621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:27:01.085988401 +0000 UTC m=+4496.644624886" watchObservedRunningTime="2026-11-03 15:27:01.086912931 +0000 UTC m=+4496.645549415" Nov 03 15:27:03.984492 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:03.984460 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tjsp9_fc214996-6ee2-4bf3-a2cd-49cd4eceaa09/global-pull-secret-syncer/0.log" Nov 03 15:27:04.991695 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:04.991074 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-frkjs_905b521a-2b6a-4fb6-ba5b-33311a2613b0/konnectivity-agent/0.log" Nov 03 15:27:05.273635 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:05.273322 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-191.ec2.internal_b01771a71144616672fedcb36938a552/haproxy/0.log" Nov 03 15:27:07.808774 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:07.808561 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5f464594c6-562pp_428b2a8c-5dc4-4b29-a535-35b8b20cf721/manager/0.log" Nov 03 15:27:08.010789 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:08.010754 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qdflj_f884e514-a7a4-459f-aba9-62f7ce76671a/manager/0.log" Nov 03 15:27:08.774696 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:08.774591 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dfdc587c5-vbgqs_fc2d0b86-48fd-48f6-89fa-150de7e46ca9/manager/0.log" Nov 03 15:27:11.075048 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:11.075015 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5f464594c6-562pp_428b2a8c-5dc4-4b29-a535-35b8b20cf721/manager/0.log" Nov 03 15:27:11.254895 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:11.254698 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qdflj_f884e514-a7a4-459f-aba9-62f7ce76671a/manager/0.log" Nov 03 15:27:11.746940 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:11.746910 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-k9lp2_bf69a67f-797a-4489-a929-789d47efc69c/kuadrant-console-plugin/0.log" Nov 03 15:27:12.129122 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:12.129051 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dfdc587c5-vbgqs_fc2d0b86-48fd-48f6-89fa-150de7e46ca9/manager/0.log" Nov 03 15:27:13.827726 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:13.827297 6389 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5"] Nov 03 15:27:13.832651 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:13.832624 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:13.854350 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:13.854320 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5"] Nov 03 15:27:13.904919 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:13.904880 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22dfae43-0106-45ea-bcc6-a8fabdf5d261-podres\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:13.905110 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:13.904932 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22dfae43-0106-45ea-bcc6-a8fabdf5d261-sys\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:13.905110 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:13.905064 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22dfae43-0106-45ea-bcc6-a8fabdf5d261-lib-modules\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:13.905234 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:13.905114 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vjwg\" (UniqueName: \"kubernetes.io/projected/22dfae43-0106-45ea-bcc6-a8fabdf5d261-kube-api-access-7vjwg\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:13.905234 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:13.905206 6389 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22dfae43-0106-45ea-bcc6-a8fabdf5d261-proc\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:14.008181 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.008145 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22dfae43-0106-45ea-bcc6-a8fabdf5d261-podres\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:14.008457 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.008419 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22dfae43-0106-45ea-bcc6-a8fabdf5d261-sys\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:14.008583 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.008492 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22dfae43-0106-45ea-bcc6-a8fabdf5d261-lib-modules\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:14.008583 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.008526 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vjwg\" (UniqueName: \"kubernetes.io/projected/22dfae43-0106-45ea-bcc6-a8fabdf5d261-kube-api-access-7vjwg\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:14.008750 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.008641 6389 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22dfae43-0106-45ea-bcc6-a8fabdf5d261-proc\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:14.008807 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.008758 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22dfae43-0106-45ea-bcc6-a8fabdf5d261-proc\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:14.008807 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.008272 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22dfae43-0106-45ea-bcc6-a8fabdf5d261-podres\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:14.008904 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.008822 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22dfae43-0106-45ea-bcc6-a8fabdf5d261-sys\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:14.008904 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.008879 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22dfae43-0106-45ea-bcc6-a8fabdf5d261-lib-modules\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:14.031308 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.031272 6389 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vjwg\" (UniqueName: \"kubernetes.io/projected/22dfae43-0106-45ea-bcc6-a8fabdf5d261-kube-api-access-7vjwg\") pod \"perf-node-gather-daemonset-j8hj5\" (UID: \"22dfae43-0106-45ea-bcc6-a8fabdf5d261\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:14.152946 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.152867 6389 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:14.475896 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.475868 6389 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5"] Nov 03 15:27:14.479121 ip-10-0-130-191 kubenswrapper[6389]: W1103 15:27:14.479091 6389 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod22dfae43_0106_45ea_bcc6_a8fabdf5d261.slice/crio-5d9395bb3958cce2fb10efc6bcaa931b2a0f002ef6a94eb99bd7d305c60cef6f WatchSource:0}: Error finding container 5d9395bb3958cce2fb10efc6bcaa931b2a0f002ef6a94eb99bd7d305c60cef6f: Status 404 returned error can't find the container with id 5d9395bb3958cce2fb10efc6bcaa931b2a0f002ef6a94eb99bd7d305c60cef6f Nov 03 15:27:14.547556 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.547527 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5f464594c6-562pp_428b2a8c-5dc4-4b29-a535-35b8b20cf721/manager/0.log" Nov 03 15:27:14.726235 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:14.726140 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qdflj_f884e514-a7a4-459f-aba9-62f7ce76671a/manager/0.log" Nov 03 15:27:15.158312 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:15.158226 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" event={"ID":"22dfae43-0106-45ea-bcc6-a8fabdf5d261","Type":"ContainerStarted","Data":"504259a14c777764d85611ab408043c85358b399411edafb183c5d637ce5e9a9"} Nov 03 15:27:15.158312 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:15.158289 6389 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" event={"ID":"22dfae43-0106-45ea-bcc6-a8fabdf5d261","Type":"ContainerStarted","Data":"5d9395bb3958cce2fb10efc6bcaa931b2a0f002ef6a94eb99bd7d305c60cef6f"} Nov 03 15:27:15.158768 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:15.158320 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:15.204124 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:15.204073 6389 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" podStartSLOduration=2.204056537 podStartE2EDuration="2.204056537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:27:15.202513729 +0000 UTC m=+4510.761150213" watchObservedRunningTime="2026-11-03 15:27:15.204056537 +0000 UTC m=+4510.762693020" Nov 03 15:27:15.302987 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:15.302960 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-6k8f7_af2b7bc6-b143-4ffd-b58c-3ca6f13eec56/kube-state-metrics/0.log" Nov 03 15:27:15.454670 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:15.454636 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-6k8f7_af2b7bc6-b143-4ffd-b58c-3ca6f13eec56/kube-rbac-proxy-main/0.log" Nov 03 15:27:15.486694 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:15.486661 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dfdc587c5-vbgqs_fc2d0b86-48fd-48f6-89fa-150de7e46ca9/manager/0.log" Nov 03 15:27:15.591628 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:15.591587 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-6k8f7_af2b7bc6-b143-4ffd-b58c-3ca6f13eec56/kube-rbac-proxy-self/0.log" Nov 03 15:27:16.518361 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:16.518332 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zbs97_b57fa4d4-1e9d-4a56-aba9-c6314ff12aea/node-exporter/0.log" Nov 03 15:27:16.661998 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:16.661956 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zbs97_b57fa4d4-1e9d-4a56-aba9-c6314ff12aea/kube-rbac-proxy/0.log" Nov 03 15:27:16.804415 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:16.804334 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zbs97_b57fa4d4-1e9d-4a56-aba9-c6314ff12aea/init-textfile/0.log" Nov 03 15:27:17.380177 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:17.380142 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jsdph_8e380172-2ed6-4bd3-9fdb-83a694b400d0/kube-rbac-proxy-main/0.log" Nov 03 15:27:17.520696 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:17.520670 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jsdph_8e380172-2ed6-4bd3-9fdb-83a694b400d0/kube-rbac-proxy-self/0.log" Nov 03 15:27:17.660811 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:17.660734 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jsdph_8e380172-2ed6-4bd3-9fdb-83a694b400d0/openshift-state-metrics/0.log" Nov 03 15:27:19.353265 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:19.353236 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65979fb8bc-sh6c8_608de1a1-c806-48bb-aecc-def971eca078/telemeter-client/0.log" Nov 03 15:27:19.503785 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:19.503757 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65979fb8bc-sh6c8_608de1a1-c806-48bb-aecc-def971eca078/reload/0.log" Nov 03 15:27:19.665915 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:19.665835 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65979fb8bc-sh6c8_608de1a1-c806-48bb-aecc-def971eca078/kube-rbac-proxy/0.log" Nov 03 15:27:21.135264 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:21.135222 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-m6wvl_ea0ab114-50e5-4549-8af4-f361d3f23df7/discovery/0.log" Nov 03 15:27:21.174581 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:21.174557 6389 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-j8hj5" Nov 03 15:27:21.245320 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:21.245290 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76667dfcdc-rj9l9_502f2df2-d66f-4023-acb6-6ef16ac7f5a8/kube-auth-proxy/0.log" Nov 03 15:27:21.480988 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:21.480960 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-7jwxs_e5733a68-00fa-4304-87d1-e10764bf1d6c/istio-proxy/0.log" Nov 03 15:27:21.703500 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:21.703469 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-dzkt9_fda036b9-bdd2-41a1-b2d5-bddc0f0e396e/istio-proxy/0.log" Nov 03 15:27:21.785974 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:21.785891 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-lnxjp_69abdd4b-9f73-4ae6-b1c8-e5100f15ad80/istio-proxy/0.log" Nov 03 15:27:22.169542 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:22.167502 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76bfcf8f7d-tzw9j_8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69/payload-pre-processing/0.log" Nov 03 15:27:24.802803 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:24.802776 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-d56c57f9b-fjg7p_4fbb10eb-9066-40b9-84c6-315b93015a50/console/0.log" Nov 03 15:27:24.997320 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:24.997284 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-dzpfq_0694b8ee-14f6-495d-a662-8f06bdfe1c66/download-server/0.log" Nov 03 15:27:25.345047 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:25.345021 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-k9lp2_bf69a67f-797a-4489-a929-789d47efc69c/kuadrant-console-plugin/0.log" Nov 03 15:27:28.379994 ip-10-0-130-191 kubenswrapper[6389]: E1103 15:27:28.379945 6389 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbbs" containerName="extractor" Nov 03 15:27:29.651104 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:29.651075 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j6lgv_977c941d-36f2-4c16-8396-95bdce627a74/dns/0.log" Nov 03 15:27:29.795912 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:29.795872 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j6lgv_977c941d-36f2-4c16-8396-95bdce627a74/kube-rbac-proxy/0.log" Nov 03 15:27:30.242644 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:30.242596 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qnhkx_b586d272-c9c0-455f-aec9-333ae6d0c69a/dns-node-resolver/0.log" Nov 03 15:27:31.031865 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:31.031827 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-m6wvl_ea0ab114-50e5-4549-8af4-f361d3f23df7/discovery/0.log" Nov 03 15:27:31.142217 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:31.142189 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76667dfcdc-rj9l9_502f2df2-d66f-4023-acb6-6ef16ac7f5a8/kube-auth-proxy/0.log" Nov 03 15:27:31.376962 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:31.376888 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-7jwxs_e5733a68-00fa-4304-87d1-e10764bf1d6c/istio-proxy/0.log" Nov 03 15:27:31.611424 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:31.611393 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-dzkt9_fda036b9-bdd2-41a1-b2d5-bddc0f0e396e/istio-proxy/0.log" Nov 03 15:27:31.698464 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:31.698439 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-lnxjp_69abdd4b-9f73-4ae6-b1c8-e5100f15ad80/istio-proxy/0.log" Nov 03 15:27:31.899499 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:31.899469 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dlwbh_edb1d4a1-0656-4417-8b7d-86a68db196ee/node-ca/0.log" Nov 03 15:27:32.076999 ip-10-0-130-191 kubenswrapper[6389]: I1103 15:27:32.076927 6389 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76bfcf8f7d-tzw9j_8e00cb7a-21c7-48a1-9c99-ca8a2e3c7c69/payload-pre-processing/0.log"